For educational purposes, I would like to write my own i2c protocol. Is it possible to write a custom i2c library to communicate using regular ports, or should I use the SCL and SDA ports? Can I mimic SDA port by using an external transistor with pull-up resistor connected to digital pin?
Is it possible, yes, and has been done many times already.
It can be done with GPIO ports.
You can mimic SDA port with an external transistor and a pull-up, but usually nobody does it because it can be done without the external transistor by simply using the GPIO ports. No external hardware at all, only the pull-up resistors that are needed anyway.
What you seem to be trying to do is just called software bit-banged I2C interface.
Since you tagged "Arduino", I would be very surprised if the Arduino framework does not already have a software based I2C implementation built-in you can simply use so you don't even have to write it yourself.
Edit: Just confirmed, Arduino comes with "SoftWire" library .