I'm trying to understand how transformers work. I understand the primary/secondary stepping down/up voltage or current, but why do they use oil (in oil tank) with transformers?
Transformers are oil filled for two reasons:
The transformer windings dissipate waste heat, which needs to be removed. Transformer oil absorbs this heat from the windings and conducts it to the outside of the transformer, where it can be dissipated to the outside air.
It is possible to build an air-cooled transformer. These are used where spillage of the insulating oil is an un-acceptable hazard. I.e. for indoor substations, where the oil would be a fire hazard. Dry-type transformers are generally more expensive than oil-filled transformers.
All parts which might be at different voltages must be insulated from each other.
In air insulated equipment (no oil) adequate clearances must be allowed around all live parts, to prevent flashover through the air. Alternatively, solid insulation would be required to all live parts (expensive.)
Oil is a much better insulator than air, so the clearances can be reduced. This makes the equipment much more compact.
Commercially produced transformers use both solid and liquid insulation. Solid paper insulation is used for the windings, where compactness is critical. Oil insulation is used to cover all the parts that don't have solid insulation. The oil also penetrates into the paper and fills the air gaps, improving the quality of the paper insulation.
Poly-chlorinated biphenyl (PCB) used to be the insulating liquid of choice. It has ideal engineering properties - it's a great insulator, high thermal conductivity, etc. Unfortunately it's also highly toxic. Like asbestos, it is now banned, and cleanup/disposal of existing PCB-filled equipment is an ongoing expense.
Most transformers are now filled with mineral oils of some type. These oils are specially formulated for the application. There are a few different types and they don't all intermix! Some care is required to select the correct oil when topping up the oil in an existing transformer.
It's also possible to fill transformers with vegetable-based oils (again, specially formulated for the application.) Claimed benefits are - less fire risk, better moisture adsorption, environmentally friendly. This costs a bit more than mineral oil, so it is usually available as a "special option", not the default.
High power transformers often have the core and windings immersed in a liquid (not always oil, poly-chlorinated byphenol used to be popular). These transformers usually have fins sticking out from the core. One purpose of the liquid is to convect heat from the core to the fins, where it can be more easily dissipated to the outside.
The liquid is also chosen to be a good dielectric. This acts as additional self-healing insulation.