27
\$\begingroup\$

Background

I am working with a client on a large project which requires a custom networking chip to be designed to solve the data transfer requirements within the project. The network is intended to send small packets a few inches from one PCB to another over a single twisted pair cable. We will be designing and specifying the network protocol, and another company will be responsible for the silicon implementation.

I estimate that 20Mbps data rate between nodes will easily cope with the amount of data that needs to be sent, with plenty of head room should the amount of data increase in the future.

Problem

The client is asking me why I am specifying only 20Mbps. Why not something like 1Gbps? Wouldn't that be better? Intuitively, I feel that cranking up the data rate massively beyond what would be needed is a bad idea. Initially, I thought that the cabling would need to be shielded (which I don't want), but looking at the Ethernet cable categories, I see that Gigabit Ethernet can run on Cat 6 cable, which doesn't need to be shielded.

Other Constraints

  • The project is desperately space constrained, and do we don't have room for things like magnetics, unless it's a very small component (0603 max).
  • The cables need to be as slim and flexible as possible.
  • The device will run from plug-in power, so there is no particular low-power requirement.

Question

What are the problems, in terms of silicon design, cabling, and anything else, that may be faced at 1Gbps, that wouldn't be nearly as bad at 20Mbps? Should I go with my client's suggestion of implementing the network at 1Gbps, or should I insist on implementing only what is required?

We're under strict NDA, so I can't give too many details about our requirements. But please leave a comment if clarification is needed.

\$\endgroup\$
  • 3
    \$\begingroup\$ If the application needs 20 Mbps throughput, then likely a 100 Mbps older standard would be sufficient, considering all overhead and such. So the question will be down to the cost of implementation, future parts availability. Did you check corresponding costs? Also, what do you mean "in terms of silicon design"? Are you planning to design your own silicon? \$\endgroup\$ – Ale..chenski May 19 '18 at 22:28
  • 9
    \$\begingroup\$ 1Gbps may be an overkill ... 100Mbps may be a more suitable next step in speed .... if the customer is concerned with future-proofing the device, maybe explore the use of data compression ...... maybe demonstrate to the customer how much data throughput is achieved with 20Mbps ... show them an HD youtube video and explain that the bandwidth is only 8Mbps or so \$\endgroup\$ – jsotola May 19 '18 at 22:29
  • 12
    \$\begingroup\$ Why do you want a "networking" protocol for a point-to-point connection? \$\endgroup\$ – The Photon May 20 '18 at 0:20
  • 20
    \$\begingroup\$ Designing "a network protocol" from scratch doesn't look like a particularly wise idea. It could be quite costly for design, implementation of physical layer, validation of physical layer, then protocol layer, all associated software and validation thereof, etc. etc. Have you heard of HSIC, a sort of USB over 1.2VLVCMOS? \$\endgroup\$ – Ale..chenski May 20 '18 at 1:44
  • 20
    \$\begingroup\$ Please, throw together a demo with 100M ethernet ICs without magnetics to demo to the user, and to yourselves, what can be done with standard off the shelf parts. Do not do custom silicon, custom protocols, without understanding why standard ones will not do. \$\endgroup\$ – Neil_UK May 20 '18 at 5:58
24
\$\begingroup\$

A few reasons:

Power

Faster speed means more power. Not only do you need faster analog circuits, which will consumer more power, all your electronics surrounding them need to be faster. Your digital systems, your latches, clock management, etc. If you get that 1 Gbps by using multilevel signalling you now need better ADCs and DACs. You might need to start dealing with more complex filtering. You could start requiring FEC which also needs to keep up.

Chip size

Faster means more going on. You need better clock stability, which means bigger circuits. You need better timing, which means a more complex clock recovery system. You might need to switch to using DSP to do channel equalization. Your potentially needed FEC needs chipspace.

Environment sensitivity

If you switch from a few tens of megabaud to whatever is needed for gigabit, you will become far more sensitive to the environment. Small mismatches which might be unnoticeable at a few tens of MHz become resonant stubs at higher frequencies. Reflections might start causing intermittent performance. A nicked cable due to abuse over the years (I don't know the application environment for your product) might be fine for lower speeds, but cause poor performance when you go higher.

Design effort

I think it is obvious from all of the additional issues I discussed above that the time and effort of designing a faster communication link is significant. This alone should be enough of a reason.

EMI

Faster speed means meeting EMI requirements could be harder.

| improve this answer | |
\$\endgroup\$
  • 5
    \$\begingroup\$ @Rocketmagnet: Also worth pointing out that actual 802.3 1000Base-T gig-E over Cat5e sends signals in both directions at once over 4 pairs of wires, as well as using a multi-level encoding, to keep the frequencies down to the same 125MHz as 100Base-T. So each receiver has to subtract what its own transmitter is sending, to get just the signal sent by the other end. \$\endgroup\$ – Peter Cordes May 21 '18 at 1:23
  • 2
    \$\begingroup\$ @PeterCordes - That's very interesting, I'll look into that. But it sounds like considerably more complexity than basic LVDS. \$\endgroup\$ – Rocketmagnet May 21 '18 at 10:01
  • 2
    \$\begingroup\$ @Rocketmagnet: yes, GigE sounds like a very poor choice for this. There's a reason GigE transceivers use more power than 100M. \$\endgroup\$ – Peter Cordes May 21 '18 at 19:37
24
\$\begingroup\$

TTL (single-ended, unterminated) signals can easily handle 20 Mbps or more — look at SPI, for example. If you're only going a few inches, ribbon cable and IDC connectors (or a backplane of some sort) will get you from board to board.

1 Gbps puts you into the realm of having to deal with impedance-controlled traces, connectors and cables. The receivers will need to use PLL/DLL techniques to maintain synchronization and separate clock/data, whereas at the slower speed, normal synchronous logic will be sufficient. The 50× overkill and the additional headaches are simply not worth it, if you're sure that 20 Mbps will suffice for the foreseeable future.


I once designed (25 or so years ago) a custom serial bus protocol for board-to-board control and status among boards in a telecom rack. Sort of a cross between I2C and SPI — unidirectional signals like SPI, but embedded device addresses like I2C.

| improve this answer | |
\$\endgroup\$
  • \$\begingroup\$ Assuming that we're already intending to use impedance controlled traces, cables and connectors, are there any other problems? \$\endgroup\$ – Rocketmagnet May 19 '18 at 22:38
  • 1
    \$\begingroup\$ The receivers will need to use PLL/DLL techniques to maintain synchronization and separate clock/data. At the slower speed, normal synchronous logic will be sufficient. \$\endgroup\$ – Dave Tweed May 19 '18 at 22:41
  • 10
    \$\begingroup\$ That depends on details about your implementation technology that you're probably not at liberty to disclose. If you are using FPGAs (as implied by your use of the tag), note that all of the major vendors have canned solutions for chip-to-chip communication at various speeds. You'd be well-advised to use that for your physical layer, and implement your custom protocol on top of it. \$\endgroup\$ – Dave Tweed May 20 '18 at 4:08
  • 1
    \$\begingroup\$ Designing a pll block is not trivial and buying the IP can easily set you back a 50-100k $ \$\endgroup\$ – Mike May 20 '18 at 6:35
  • 1
    \$\begingroup\$ @Tustique You think it is trivial to design your own PLL on silicon? \$\endgroup\$ – user253751 May 21 '18 at 3:06
14
\$\begingroup\$

The obvious question is, "Does 1 Gbps mean 1000BASET Ethernet?" If that's what the customer is thinking, your requirement that, "we don't have room for things like magnetics" rules that out right away. Ethernet does use magnetics on the physical layer, and when I designed an interface some years ago the magnetics were part of a roughly 1 inch cube.

You say you're using FPGAs, but you don't say whose. If you're going with Xilinx, you should be aware that the current models natively support LVDS, which would seem ideal for your purpose. Early LVDS systems (hi-def televisions) ran at 122 Mbps, and the technology can go well over a Gbps if you really need to. Being differential, and assuming your two boards are not using wildly floating grounds, noise immunity is excellent.

As for your specific choice of clock frequencies, adding more headroom than you think you need is one of those decisions which can save your bacon in the future, so I wouldn't rule out picking something like 100 MHz, but that's up to you. You might acquaint your customer with Roberge's Law (Jim Roberge was a well-known electrical engineering professor at MIT a few decades ago): "Those who ask for more bandwidth than they need deserve what they get." Granted, he was talking about servo systems, but the principle remains good over a remarkably wide range of disciplines.

| improve this answer | |
\$\endgroup\$
  • 7
    \$\begingroup\$ Ethernet does not need the magnetics if you are making a non-standard connection. It is quite reliable if implemented without the isolation. \$\endgroup\$ – Jack Creasey May 20 '18 at 4:10
  • 1
    \$\begingroup\$ @JackCreasey but can one then still call it Ethernet? \$\endgroup\$ – Mels May 20 '18 at 10:22
  • 2
    \$\begingroup\$ It is certainly not IEEE 802.3 compliant. It won't connect to a compliant end point, but I did say it would be non-compliant without the magnetics. I doubt it would pass the test suite for signals on the backplane connection, but it should pass everything else including jitter. I'd still call it Ethernet, but you may not. \$\endgroup\$ – Jack Creasey May 20 '18 at 14:16
  • 2
    \$\begingroup\$ FYI: the only Google result for "Those who ask for more bandwidth than they need deserve what they get." is this answer. \$\endgroup\$ – user253751 May 21 '18 at 22:48
11
\$\begingroup\$

The application that you describe does not make sense to jump right into a custom silicon solution. The data rates that you anticipate can be easily handled by moderately priced FPGA technology and the FPGA can be programmed to implement the special protocol if your really believe that such protocol is needed.

Much more often you should be considering a standard physical layer and then build the customized protocol on top of that. For a net comm channel bandwidth of 20Mbps you should plan for a protocol overhead of some amount because framing, error check coding and synchronization eat up some of your bandwidth. So maybe consider a higher raw bandwidth to accommodate this overhead.

Once you have your design proven out then you can go to the FPGA vendor and get them to produce a hard chip design from the FPGA programming. This approach mitigates all early development risk and lowers overall NRE costs a tremendous amount over the "dive into custom silicon just because it seems cool".

| improve this answer | |
\$\endgroup\$
  • \$\begingroup\$ That's exactly the route we intend to take. \$\endgroup\$ – Rocketmagnet May 21 '18 at 10:02
11
\$\begingroup\$

The actual question is, why to design a protocol when everything already exists.

For Ethenet solutions you take 10/100 and not 1GbE because it's still a little cheaper and much easier to layout. By the way, Ethernet can work without magnetics. But it does require MAC, which can be extra IC. Or do you have one in a microcontroller?

20Mbps is something that fits Rs485 or such layer, which is even cheaper and simpler. Twisted pairs come with all kinds of cables, more or less flexible, with connectors or just soldered to your PCB.

Ah, most importantly. It's easier to screw up with 1Gb. But if they need room for further growth, it limits less.

Bottom line: you need to understand your system requirements.

| improve this answer | |
\$\endgroup\$
10
\$\begingroup\$

I'd suggest the simplest route with the best likelihood of success and least software overhead would be to implement a 100Mbps Ethernet connection. You can implement this without any magnetics involved when the distances are small.

Here's a start with information on the Intel 8255 PCI-Ethernet controller, and an application note on connections without the magnetics.
I'm not suggesting you use the 8255, but you can get IP (10/100/1000Mbps) for any of the FPGAs you are likely to use very easily, and it's well debugged.

Assuming you have a processor in the mix, supporting a standard Ethernet controller is a very low software effort way to implement point to point networking.
We used a bunch of this type of point to point connections on specialized motherboards at Intel, they were easy to debug, and very reliable.

| improve this answer | |
\$\endgroup\$
  • 4
    \$\begingroup\$ Although ethernet could do the protocol, 100 Base ethernet has a minimum distance requirement of 1 metre (this bit me some years ago using an ethernet over backplane implementation). The OP mentioned the distance was a few inches which rather rules out ethernet at the physical layer. \$\endgroup\$ – Peter Smith May 20 '18 at 12:32
  • 1
    \$\begingroup\$ @PeterSmith, the minimum distance DOES NOT apply to Ethernet without the magnetics. This will work down to just a few inches. \$\endgroup\$ – Jack Creasey May 20 '18 at 14:07
5
\$\begingroup\$

The answers here are technical, i give a requirements engineering perspective:

My view on it is simple

  • You need at least 20Mbps to make it work, so dont specify 20 but "20 or more" for the application.

  • any faster hardware also fulfills your requirement

  • if the faster HW is cheaper/easier to develop because of existing standards, then your requirement can be fulfilled by these ones, too.

  • If the customer wants more, try to figure out if there is something behind it (it could be that they already plan upgrades and want to stay compatible between boards when swapping)

| improve this answer | |
\$\endgroup\$
3
\$\begingroup\$

Power, signal integrity, and timing. I worked on a chip with a 25 gbps interface and that meant a clock rate of 1.6 GHz and a ton of power. If we could have run at 19.2 the clock rate would have been 1.2 GHz. Greater than 200ps of extra margin per clock period, that would have been a huge help.

I've never done board design but I expect that 20 Mbps is no problem. 1 Gbps is still not that tough but a lot harder than 20 Mbps.

| improve this answer | |
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.