In the Gigabit Ethernet world, the media access controller (MAC) communicates with the physical layer chip (PHY) through the Gigabit Medium-Independent Interface (GMII)
The GMII is an 8-bit-wide interface carrying 1000 Mb/s. So its clock rate is 125 MHz.
The Physical Coding Sublayer (PCS) within the PHY performs the 8b/10b encoding. So its output rate is 1250 Mb/s.
The interface between the PCS and the Physical Medium Attachment (PMA) sublayer is, in the systems I'm familiar with, internal to a chip, so it might be 10-bit-wide parallel with a 125 MHz clock, but it might also be 20, 40, or 80 bits wide with a corresponding lower clock rate.
The Physical Medium Dependent (PMD) sublayer is mainly embodied in what we normally think of as the fiber optic transceiver. The interface between the PMA and PMD is (again, in the systems I know of) a 1250 Mb/s serial interface. The techniques to implement this interface on a PCB, including several inches of trace between the PHY chip and the transceiver, have been established for over 15 years.
but in the case of 10Gbps fiber transmissions, what is changed? I do not think it will come to use a frequency of such a high clock.
It is in fact possible to route 10 Gb/s signals over a PCB, and many 10 Gb/s fiber optic transceivers use a serial interface. This does require careful choice of PCB materials, and equalization techniques may be used at one or both ends of the trace.
The serial baud rate is only 10.3125 Gbaud, rather than the 12.5 Gbaud that would be needed for 8b/10b encoding, because 10 Gigabit Ethernet uses the more efficient 64b/66b encoding instead.
The optical signals at 10 Gb/s (as at 1 Gb/s) are typically simple on-off keying (OOK) which just means sending a high powered optical pulse for a '1' and a low power (ideally zero power) pulse for a '0'. Zero power for the '0' is typically not achievable because it takes a laser much longer to turn on after it is fully turned off than it does to turn it brighter when it has been only weakly on.
There is one 10 Gigabit Ethernet PMD standard, 10GBASE-LX4 that uses 4 wavelength division multiplexed channels, each carrying a 3.125 Gb/s OOK signal to provide a 12.5 Gbaud 8b/10b encoded 10 Gb/s.