4
\$\begingroup\$

I have an application that requires optical transmission of 4x16 Gbps SERDES lanes over fiber. A QSFP28 module for 100G ethernet seems like a great commercial solution--it can take the four lanes and combine them over a single fiber channel. The only hitch is that all of the QSFP28 modules I can find are only characterized over a narrow electrical lane rate, about 25-27 Gbps.

Here's an example from FS, but it's a similar story for all vendors I've checked: https://www.fs.com/products/75309.html Snippet from QSFP28 datasheet

I understand why the datasheet recommended range is so limited, since they are designed for a singular networking application. However, my gut also says that a lower but not-too-low lane rate like 16 Gbps would be okay for the internal SERDES transceivers. It would also enable the modules to be used in 40G/4x10G applications, so I'm wondering if there are any resources that would serve as evidence to this point.


Additional Research

I accepted Joren's answer since it showed quite clearly that the limitations of the CDR/retiming functionality of QSFP28 modules would be the foremost barrier. I also did a little research that I want to share here. There is a workaround provided in the QSFP28 memory map. See the snippet below from SFF-8636: Management Interface for 4-lane Modules and Cables, which shows that it is possible to disable retiming functionality. This presumably comes at the expense of signal integrity at the receiving end and consequent maximum link length.

I'll probably run some tests, and also take advice to look at alternate fiber specifications.

Snippet from SFF-8636

\$\endgroup\$
6
  • 1
    \$\begingroup\$ Look for transceivers designed for Fibre Channel applications rather than Ethernet. They're more likely to have rates at multiples of 8. But I don't recall if 64 Gbps aggregate is a typical Fibre Channel application. \$\endgroup\$
    – The Photon
    Commented Jun 15, 2020 at 14:44
  • \$\begingroup\$ @ThePhoton thank you, that also sounds promising. Any recommended vendors that I can start with? \$\endgroup\$
    – jalalipop
    Commented Jun 15, 2020 at 15:14
  • \$\begingroup\$ Sorry I just looked it up and 14.025 Gbaud is the closest Fibre Channel rate to what you want. They call it "16G Fibre Channel" due to some creative accounting when they switched from 8b10b encoding to 64b66b. \$\endgroup\$
    – The Photon
    Commented Jun 15, 2020 at 15:54
  • \$\begingroup\$ Oh actually my quoted lane rates are after 64b66b encoding. FWIW I did some research based on your comment and did find some fiber channel 150G CXP modules that run down to 1G, also with no retiming and lower lane rates. It seems that I can either work my lane rates into a standardized spec, or sacrifice retiming, or both. This sets up some interesting trades in my application... \$\endgroup\$
    – jalalipop
    Commented Jun 15, 2020 at 16:03
  • 1
    \$\begingroup\$ Unfortunately the way FibreChannel names their rates is according to "what would the baud rate be if we used 8b10b encoding, regardless of what the actual encoding is". So when they say "1G" they mean 1 Gbaud line rate with 8b10b encoding. But when they say "16G" they mean 16x the data throughput of the 1G version, but the line rate is only 14.xxx Gbaud due to switching to 64b66b encoding. \$\endgroup\$
    – The Photon
    Commented Jun 15, 2020 at 17:16

2 Answers 2

5
\$\begingroup\$

The problem is often in clock-recovery circuits. The CDR chips used in these kinds of modules have PLLs/recovery loops with limited locking rates. An example (but made for QFSP40) is the Semtech GN2405A. As a result, you can only use them with symbol rates that match them. Sometimes, as is the case with the Semtech IC I linked to, you can have modules that can lock to lower rates too, but again only fixed windows. Sometimes you can also disable the retiming circuits of such a chip. However, all of this stuff is controlled by the microcontroller integrated in the QSFP module, and is not something you have access to as user.

\$\endgroup\$
4
  • \$\begingroup\$ This is very helpful, especially seeing the Semtech part. So a limiting or linear interface might not be an issue, but this looks like a definite limitation for re-timed interface. I've been spoiled by FPGA transceivers with huge frequency ranges! \$\endgroup\$
    – jalalipop
    Commented Jun 15, 2020 at 14:32
  • 1
    \$\begingroup\$ Joren, thanks again, this set me on an enlightening path. I may be in luck, it seems the QSFP28 standard supports disabling retiming for this exact use-case. I'll update my question with some additional resources. \$\endgroup\$
    – jalalipop
    Commented Jun 15, 2020 at 15:04
  • \$\begingroup\$ @jalalipop I wasn't aware that was part of the QSFP28 standard. I know I've seen it implemented on the CDRs but I wasn't aware of it being accessible by standard. \$\endgroup\$
    – Joren Vaes
    Commented Jun 15, 2020 at 15:05
  • \$\begingroup\$ it does appear to be an optional register, so jury is still out on whether my vendor of choice supports it... \$\endgroup\$
    – jalalipop
    Commented Jun 15, 2020 at 17:08
4
\$\begingroup\$

If you aren't running at a compatible line rate, you'll need to disable the transceiver CDR circuitry. I have had to do this before when running at rates other than 25G, including at 10G. This can be done via writing to a specific MSA register via the I2C interface. The register is number 98, with the lower nibble controlling the CDR on the four RX channels, and the upper nibble controlling the CDR on the four TX channels. So try writing 0 to this register and see what happens. I will note that not all modules have CDRs, and not all modules implement this register correctly.

\$\endgroup\$
4
  • \$\begingroup\$ +1, I was able to come to the same conclusion by looking at the memory map spec. However, it lists register 98 as optional. Is it commonly implemented? I've reached out to FS to get one datapoint. \$\endgroup\$
    – jalalipop
    Commented Jun 15, 2020 at 18:21
  • \$\begingroup\$ It seems to be a bit hit or miss. Asking the mfr is probably a good idea. \$\endgroup\$ Commented Jun 15, 2020 at 18:22
  • 1
    \$\begingroup\$ Also, I should mention that the first time I ran in to this issue was with a transceiver from innolight that did not mention CDR in the datasheet at all, but register 98 was implemented and was able to disable the CDR properly. Also, it worked at 12.890625 Gbps with the CDR enabled. \$\endgroup\$ Commented Jun 15, 2020 at 18:41
  • 1
    \$\begingroup\$ Just to add that register 194 has bit 7 Tx CDR On/Off Control implemented. 1 if controllable, else 0. and bit 6 Rx CDR On/Off Control implemented. 1 if controllable, else 0. to indicate if the module implements the CDR On/Off controls in register 98. \$\endgroup\$ Commented Dec 7 at 23:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.