4
\$\begingroup\$

I have a Gigabit Ethernet port and currently I try to troubleshoot an asymmetry in data throughput. Doing iperf3 testing I notice that sending is at 900 MBit/s and receiving at 400 MBit/s. During that troubleshoot I have captured waveforms right behind the magnetics.

What troubles me the most is, that the waveforms, when the interface is set to 1000Base-T are not so clean compared to 100Base-TX mode. On the other hand, if looking at the Application Note "Physical Layer Compliance Testing for 1000BASE-T Ethernet" from Tektronix (link), the waveform depicted there is quite "messy" too. Maybe I should do some measurements acoording to that document (found while posting this question).

Is my measurement affecting the signals? I do not see any degradation in speed when the probes are connected.

Does anyone have some hints regarding that assymetrical performance? 1000Base-T uses four signal pairs in full duplex mode. Two pairs are used for sending and the other two for receiving. If so, the pair used for reception could be the one with worse signal integrity, right?

100Base-TX 1000Base-T

\$\endgroup\$
5
  • 2
    \$\begingroup\$ In addition to what Justme said can you check the Ethernet port statistics in the OSes? If were signal integrity issues leading to packet drops would expect error counts (e.g rx_crc_errors) to increase during the test. \$\endgroup\$ Commented Jun 13 at 20:19
  • \$\begingroup\$ ethtool does not report any crc_errors as I remember. I need to have a closer look. Thanks. \$\endgroup\$
    – stowoda
    Commented Jun 14 at 7:11
  • 1
    \$\begingroup\$ At just 1Gsps, you can't really expect to see a very "nice" waveform for gigabit. It's a five-level amplitude modulation at 125M symbols per second, so there's a lot going on even before you mix the other direction in and consider aliasing. What is the bandwidth of your scope and probe? \$\endgroup\$
    – TooTea
    Commented Jun 14 at 7:23
  • \$\begingroup\$ Allright, seems to be not a straigt forward signal. The scope's BW is 8GHz. \$\endgroup\$
    – stowoda
    Commented Jun 15 at 12:04
  • \$\begingroup\$ Can you identify the cause by swapping cables? Or is the goal here to understand the signalling side? \$\endgroup\$
    – Criggie
    Commented Jun 16 at 6:29

2 Answers 2

13
\$\begingroup\$

You are mistaken how 1000Base-T works. There is not two receive and two transmit pairs. There are four pairs that transmit and receive simultaneously in full duplex. If there is asymmetry it can't be due to bad signal, it would affect both ends or the link could just drop out etc. The asymmetric rate is likely in the sofware or network stack. You aren't even telling what OSes run the iperf3, Windows, Linux, MacOS, FreeBSD or some other BSD.

So in 100Base-TX mode, there is one transmit pair and one receive pair, which uses MLT-3 coding (sort of special version of PAM-3). The waveforms are clean just like in your picture.

In 1000Base-T mode, there are four bidirectional pairs, each end transmitting and receiving simultaneously on all pairs, both ends using PAM-5 so there is multiple superimposed voltage levels possible on wire, not just 5. The waveforms are messy, like in your picture. To test the waveforms, usually the interface is put into test mode which transmits special patterns in sequence to see them without link partner.

\$\endgroup\$
5
  • \$\begingroup\$ Allright, so it is not a full duplex connection. Cant explain the assymetry, right. Thanks. I will put the PHY into test mode as suggested and do some measurements to gain confidence in signal integrity. I am sending from a Linux machine to a Linux machine. Another curiousity is the fact, that the throughput gets symmetric and hits 1GB/s wehen running iperf3 with udp packets. \$\endgroup\$
    – stowoda
    Commented Jun 14 at 7:08
  • 4
    \$\begingroup\$ It is full duplex. Each end subtracts their own signal to observe whats on the wire. As pointed out a problem will lead to problems in both directions. \$\endgroup\$
    – vidarlo
    Commented Jun 14 at 7:13
  • \$\begingroup\$ @stowoda It's likely some of the TCP offloading options not being enabled or not working, so you're bottlenecked by the CPU or interrupts on one of the machines. Almost certainly not a physical-layer issue but a SW one. \$\endgroup\$
    – TooTea
    Commented Jun 14 at 7:18
  • 2
    \$\begingroup\$ @stowoda It is full duplex. Nobody said it is not full duplex but the opposite, that it is full duplex. \$\endgroup\$
    – Justme
    Commented Jun 14 at 7:52
  • \$\begingroup\$ Ok, I got it now. PAM-5 is the key. Thx. \$\endgroup\$
    – stowoda
    Commented Jun 19 at 8:56
3
\$\begingroup\$

For throughput testing consider sending receiving raw Ethernet packets, and bypassing the network stack. While not communication between two Linux machines, for environmental testing of network switches have used the following techniques:

  1. switch_test_main.c uses PCAP. A single thread can achieve the full-duplex bandwidth of 1000 Mb Ethernet port. PCAP uses interrupts and did suffer packet drops when tried using the full-duplex bandwidth of a 10 Gb Ethernet port.
  2. dpdk_switch_test.c uses DPDK and a single thread can achieve the full-duplex bandwidth of a 10 Gb Ethernet port. DPDK is poll-mode access to the Ethernet hardware and doesn't use interrupts, which allows higher bandwidth without packet loss.

My above code doesn't directly allow the replacement of iperf for the bandwidth measurements, but may serve an example of how to bypass overheads of using the network stack for maximising the achieved bandwidth of Ethernet ports for hardware tests.

\$\endgroup\$
2
  • \$\begingroup\$ Thanks. Ill give it a try. \$\endgroup\$
    – stowoda
    Commented Jun 19 at 8:57
  • \$\begingroup\$ Too bad, Id need static libaries to compile your switch test for aarch64 :/ I think I am not skilled enough to doo that. \$\endgroup\$
    – stowoda
    Commented Jun 19 at 10:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.