Keep in mind that the type of ethernet that uses magnetics requires the transformer coupling in the spec. The transformers do some nice things for you, like completely decoupling common mode signals, providing significant voltage isolation, and in some cases projecting a different impedance to the electronics than what is natively on the cable.
If your whole device is floating or the connection is short with a common ground, you can present the correct impedance and voltage levels, and can ignore common mode signals, then you can in theory eliminate the transformers.
This won't work with some PHYs, since they require center tapped windings on their side. It's not uncommon for the PHY to drive the transmit pair by having the center tapped tied to power with only low side switches on each end. That effectively allows for bipolar drive with single ended switches, and is a legitimate architecture since a transformer is supposed to be there anyway. Look carefully at ethernet transformers, and you will see many offerings with center taps partly for this reason.
If you're careful, you can put just one transformer between two devices if they are physically close. Each end is still transformer coupled, so common mode rejection is there and the center tap drive trick still works. I actually did this on a small system once where both ethernet devices were contained in the same physical unit.