0
\$\begingroup\$

I'm trying to make a cluster with Zynq-7010 boards for a real-time application. One of them will be the master, and it will control eight client boards. The master board will also collect the data from clients. I tried to use 100Mbit ethernet connection to connect the nodes but it was not as fast as I need. In addition, it was not deterministic because of switch's indeterministic behavior. Could you give me some idea about how to connect them fast and deterministic way? Edit: Each node has RS-485 interface to outside. Nodes make RS-485 communication up to 10Mbit/s. The master node has to collect 500byte data from each of them every millisecond. The timestamps of each node must be identical (+-2us). The nodes must start to reply the master in 20us.

\$\endgroup\$
  • \$\begingroup\$ Use one of the PMOD connectors to send a sync pulse from the master to the slaves, and trigger the slave process off that. Then use the Ethernet to communicate the process results back to the master. \$\endgroup\$ – WhatRoughBeast Mar 5 '16 at 0:06
  • \$\begingroup\$ You need to put some numbers to your requirements. What is the nature of the communications between the boards? How much data throughput do you need, and how tightly synchronized do they need to be? Be specific. \$\endgroup\$ – Dave Tweed Mar 5 '16 at 5:37
  • \$\begingroup\$ Each node has RS-485 interface to outside. Nodes make RS-485 communication up to 10Mbit/s. The master node has to collect 500byte data from each of them every millisecond. The timestamps of each node must be identical (+-2us). The nodes must start to reply the master in 20us. \$\endgroup\$ – zontragon Mar 5 '16 at 8:44
  • \$\begingroup\$ Those boards seem to have a bunch of LVDS lines, and should be plenty fast. \$\endgroup\$ – apalopohapa Mar 5 '16 at 10:03
  • \$\begingroup\$ I agree with @WhatRoughBeast. you need a sync signal. \$\endgroup\$ – johnnymopo Mar 5 '16 at 15:32
1
\$\begingroup\$

Basically, you're probably out of luck. A bit more information might help. When a slave reports to the master, what is the packet size? Your 20 usec figure for data latency suggests a maximum of 10 bytes or less per packet, and if that is so you're just hosed.

Start with basic requirements. 500 bytes/msec times 1000 msec/second times 8 slave units gives a basic data rate to the master of 4 MB/sec, or 32 Mbit/sec. Obviously, a 10 Mbit/sec RS485 bus will not do the job.

Will 100 Mbit Ethernet work? Probably not. You need to understand the overhead requirements first. Assuming UDP, each packet of data (up to 1518 max) must be accompanied by the following header fields:

12 bytes - Interframe gap 8 bytes - Preamble and SFG 14 bytes - Ethernet header 4 bytes - S-tag 4 bytes - C-tag 20 bytes - IP header 8 bytes - UDP header 4 bytes - CRC 74 bytes total

Note that this requirement applies to each and every packet sent by each and every slave.

So, if you're trying to send packets of 10 bytes, each packet will actually contain a minimum of 84 bytes. And you can't invoke a custom, minimalist header structure, either. Since the master has to service 8 slaves a router or switch will be required, and this unit will only respond to a conventional protocol.

You have not specified exactly what the data timing represents. A 500 kB data rate combined with a 2 usec timestamp requirement suggests something like a 500 kHz/1 byte per sample or a 250 kHz/2 bytes per sample acquisition rate, but this is entirely conjecture on my part. And, as I've stated, your 20 usec latency requirement suggests that you want a small packet size. But what if you allow a large packet size, like 500 bytes/packet?

What might work is the use of a staggered sync pulse arrangement. You would provide 9 lines from the master to the slaves. The first would be an acquisition clock running at 500 kHz or 250 kHz (assuming my hunch is correct) which would be broadcast to all slaves. Since I'm assuming relatively short (say, less than 10 meters) between slaves, the sychronicity of the clock at the slaves would be much less than your 2 usec. The other 8 lines are sync pulses with a 1 kHz frequency, but spaced at 125 usec. Each sync pulse goes to one slave. Each slave uses a double-buffered data storage structure, and upon receipt of a sync pulse transmits a 500-byte data packet to the master using 100 MHz Ethernet. while simultaneously acquiring the next packet's data. The timing for each data sample is implicit in the acquisition clock rate and the location of the sample within the packet, with each packet having a known offset - some multiple of 125 usec. Since the total Ethernet packet is 574 bytes, the overhead will not interfere with the throughput requirement.

| improve this answer | |
\$\endgroup\$
0
\$\begingroup\$

I will assume that as you have nodes, they are far apart. You could provide a GPSDO (gps disciplined oscillator) and pps at your nodes. In this manner, you can provide in logic a second and clock tick counter for accurate timestamping. If you are in a building, you would need a cable drop from antennas. Then your accuracy is how good your oscillator is. In general, most clocks can be set up to be very accurate over a second or two, but if you need large holdover (GPS/sync outage) accuracy, the longer and how accurate dictate how much cost will be. For large holdover, rubidium clocks are very nice.

| improve this answer | |
\$\endgroup\$
  • \$\begingroup\$ Actually they are located in same place. \$\endgroup\$ – zontragon Mar 5 '16 at 15:50
  • \$\begingroup\$ I used to use a pc to manage all that work. But I decided to use single board computers. Because they become cheaper and fast enough. \$\endgroup\$ – zontragon Mar 5 '16 at 15:54
  • \$\begingroup\$ Then I would investigate the master providing a sync every so often, say 1 PPS. Have you verified the master can handle the amount of data you have? \$\endgroup\$ – johnnymopo Mar 5 '16 at 15:54
  • \$\begingroup\$ I ask as Im curious what 2 ARMs at 666 MHz can do \$\endgroup\$ – johnnymopo Mar 5 '16 at 15:55
  • \$\begingroup\$ 100Mbit Ethernet creates bottleneck so that I got stuck about how to collect data. Actually That's why I've posted this question. \$\endgroup\$ – zontragon Mar 5 '16 at 15:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.