22
\$\begingroup\$

10 gigabit Ethernet means that 10 billion bits are transmitted every second, but I don't understand how this is physically possible (let alone 100G Ethernet). The fastest CPUs today only run at ~8GHz, but even if transmission doesn't require a CPU, it still seems problematic.

At 10G, each bit lasts only 100 picoseconds, and in that time span, I would think that gate delays become an issue. It's not as simple as setting a line high or low for each bit, surely hundreds of transistors are required in order to output the complicated Ethernet waveform.

This seems like even more of a problem on the receiving end, since the waveform must be sampled at a very high rate, and if this uses ADCs, that introduces even more delay.

\$\endgroup\$

closed as too broad by Chris Stratton, brhans, laptop2d, Andy aka, PeterJ Jun 22 '17 at 13:52

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 6
    \$\begingroup\$ I guess the best bet would be to go and look through technical papers? \$\endgroup\$ – Eugene Sh. Jun 21 '17 at 21:27
  • 6
    \$\begingroup\$ Have a look at how the data is encoded, hint: Using PAM16 "magically" makes you need much less transitions. \$\endgroup\$ – PlasmaHH Jun 21 '17 at 21:28
  • 10
    \$\begingroup\$ 8GHz * 64bit = 512Gbps. So really 10Gbps is nowhere near as fast as the CPU. \$\endgroup\$ – Tom Carpenter Jun 21 '17 at 21:30
  • 5
    \$\begingroup\$ Most of your assumptions about how it works are quite wrong - if it worked as you imagine, it probably wouldn't be possible. But it doesn't, and it is. \$\endgroup\$ – Chris Stratton Jun 21 '17 at 21:30
  • 8
    \$\begingroup\$ @ChrisStratton I disagree that this is off-topic. We have plenty of "How does X work?" questions in the archives, and this one is reasonably narrow. \$\endgroup\$ – Adam Haun Jun 21 '17 at 21:52
25
\$\begingroup\$

Chasing this answer down took a few different links, but it appears to boil down to this:
1. 4 differential pairs (8 wires total, but only 4 lanes).
2. 800 Mega Symbols a second.
3. Using PAM16, 16 symbols are used which translates into 4 bits per baud per lane.

Given that information you come up with 4 bits*800 Mhz*4 lanes which results in 12800 Mb/s or 12.8 Gb/s. Due to encoding of error correction and other overhead, they only expect you to get 10 Gb/s from it.

Notice that the wires themselves are only changing symbols or amplitude at a frequency of 800 MHz. That's pretty meh, in terms of switching speeds for transistors.


Now that's all for 10Gb Ethernet. How they do it for 100Gb Ethernet is a little more mind boggling. For that, it appears that they actually do pump up the frequency to 10.3 GHz or 25GHz. WHAT THE HECK? See here for that table. The difference in frequency is due to how many copper pair data lanes you choose to have. Whether or not anyone has actually made this 25GHz ethernet in copper would be interesting to know. It's possible that they've only spec'ed it out. When you start getting to those frequencies, your cables either need to be really short or you just switch over to fiber optics where you can send hundreds of beams of light down a single fiber. In that way, you don't have to go at crazy speeds, you just parallelize your data at the source and deparallelize it at the destination.

References if you want to look into it more: https://en.wikipedia.org/wiki/10_Gigabit_Ethernet#Copper
https://en.wikipedia.org/wiki/Pulse-amplitude_modulation
http://www.cablinginstall.com/articles/print/volume-15/issue-7/features/technology/twisted-pair-options-for-10-gigabit-ethernet.html
https://en.wikipedia.org/wiki/Differential_signaling

\$\endgroup\$
  • 1
    \$\begingroup\$ @HaLailahHaZeh Even if they haven't found a way to encode away that effective frequency. 12.8 GHz isn't unheard of on communication channels between CPUs in a system. \$\endgroup\$ – horta Jun 21 '17 at 22:17
  • 2
    \$\begingroup\$ @HaLailahHaZeh BTW, your comment above goes against that document you linked: the modulation rate is 800Mbaud, therefore the Nyquist frequency is 400 MHz. You don't multiply the frequency by the number of voltage levels. This means that cabling capable of carrying frequencies up to ~500 MHz will suffice. \$\endgroup\$ – uint128_t Jun 21 '17 at 23:26
  • 1
    \$\begingroup\$ @HaLailahHaZeh 16 separate voltages is 4 bits, not 16 bits. \$\endgroup\$ – immibis Jun 21 '17 at 23:42
  • 2
    \$\begingroup\$ Dammit. I checked all the -CX4 & -KX4 types before posting. Didn't realize there was a 10GBase-T. \$\endgroup\$ – The Photon Jun 21 '17 at 23:43
  • 1
    \$\begingroup\$ @HaLailahHaZeh: PAM-16 does use 16 separate voltages, but only one voltage is transmitted during each symbol-time. With 10GBASE-T, the 800 MSymbols/sec means the voltage of each wire changes every 1.25 nanoseconds. Thanks for the whitepaper; it is helpful. \$\endgroup\$ – davidcary Jun 22 '17 at 12:48
14
\$\begingroup\$

10G ethernet (as described by other answers) does not do signal transitions at 10 GHz, it uses multiple level encoding spread across 4 pairs to achieve 10 Gb/s.

However, 10+ gigabit serial transceivers are quite common on high speed chips. For instance PCIe, USB3.1, thunderbolt, and similar protocols all use 10 gbit/s serial rate on individual pairs.

You are correct that "bulk" logic can't keep up with that data rate. Certainly CPU cores don't operate at that frequency, but even the logic that implements things like PCIe interfaces can't operate at that speed. Instead they use dedicated high speed SERDESs.

Data is routed with the IC in wide parallel buses. A dedicated piece of hardware does the serial-to-parallel or parallel-to-serial conversion right near the input/output. The SERDES does an absolute minimum of actual logic. Transmitters are very simple. It will have a PLL to generate the high speed serial data clock and the parallel to serial logic. Receivers are more complicated, they need to do clock recovery on the incoming data, and also framing detection to make sure that the bits are grouped properly. All in all, only a tiny bit of logic has to operate at the ultra-high speed. Yes, the propagation delays through transitions are extremely important, and the circuit has to be carefully designed so that all the signals line up properly.

\$\endgroup\$
  • \$\begingroup\$ And Thunderbolt only works 20-40 gbit/s on thin copper wires because the cables are active to amplify the weak signals and as such, quite expensive. \$\endgroup\$ – chx Jun 21 '17 at 23:06
2
\$\begingroup\$

Lots of 10 GBit/s Ethernet links are actually optical (e.g. 10GBASE-SR or 10GBASE-LR, see https://en.wikipedia.org/wiki/10_Gigabit_Ethernet) although there is also 10GBASE-T over twisted pair cables with 8P8C ('RJ45') connectors as described by @horta. As far as I know is this quite power hungry compared to the optical variants.

Transfer of data from the CPU (or rather memory) to the Ethernet card happens usually through the PCIe bus in an x86 based computer. PCIe Gen 1 lanes have a useable data transfer rate of 2 Gbit/s second (after the 8/10 bit encoding). With 8 lanes the theoretical maximum is 16 GBit/s (per direction), sufficient to drive a single port of 10 GBit/s Ethernet.

The CPU deposits the data to be transmitted in the RAM and then instructs the network card where to pick up (DMA) and similarly for reception the CPU allocates buffers and informs the network card about it when then typically generates an interrupt when the buffer(s) were filled. Note that the bandwidth to RAM is typically much larger than the one of the PCIe bus.

Today we have PCIe Gen 3 widely available which has useable data rate of roughly 8 GBit/s per lane and direction. A 16 lane slot can theoretically handle 128 GBit/s, sufficient for 100 GBit/s Ethernet (PCIe Gen 4 has been announced officially recently).

So the 'trick' to achieve high throughput inside the PC (without having to go to exorbitant signalling speeds) is to use parallel buses (RAM) or multiple serial lanes (PCIe).

For 100 Gbit/s Ethernet one typically has four links with 25 GBaud signalling speed (100GBASE-SR4, 100GBASE-LR4, 100GBASE-CR4), there are also a standards for cables with ten links (e.g. fibre pairs) of 10 Gbit/s (100GBASE-CR10, 100GBASE-SR10, 100GBASE-CR10). For longer distance links, there are also standards using only a single fibre, either using four wavelengths (100GBASE-CWDM4) or using two polarization modes and QPSK (100GBASE-ZR).

For extremely high link speeds over long haul links (such as the Marea transatlantic cable with 20 Terabit/s per fibre pair) one packs as many transmitters at different wavelengths as possible into the useable wavelength band of the fibres and amplifiers, also known as Dense Wavelength Division Multiplexing (DWDM). Note that such a multiplexer/demultiplexer is typically an optical only device at its core and is fed by multiple lower bandwidth streams which can be processed electronically in parallel.

To achieve 20 TBit/s one then also has use advanced modulation techniques where at each clock cycle multiple amplitudes and phases can be transmitted (I have seen 64QAM in a whitepaper) therefore transmitting multiple bits per clock cycle, similar to the 10GBASE-T standard described by @horta.

\$\endgroup\$

Not the answer you're looking for? Browse other questions tagged or ask your own question.