1
\$\begingroup\$

We have a distributed sensor network (12 nodes), using Ethernet as the backend protocol. These are networked at 100Mb/s to a single 1Gb/s link using a 16-port switch.

I've previously seen Java and C code to listen to a port. However our aim is to get data into the hard drive ASAP. The nodes will be transmitting with >50% network utilization meaning the Gb/s link will have a very high utilization. The data throughput will be too high to put into local RAM without issues write-read management.

My question is, what issues do I need to access when piping data as fast as possible to a hard drive. We are a single remit, that of real time data acquisition. Therefore filling RAM and later dump to HD with a gap in data acquisition is unacceptable.

Thanks.

\$\endgroup\$
1
  • \$\begingroup\$ Architectural consideration: keep it simple. A single thread can be faster than constant context switches. See martinfowler.com/articles/lmax.html for a related project: high speed financial transactions in Java. \$\endgroup\$
    – pjc50
    Commented May 2, 2015 at 13:44

2 Answers 2

1
\$\begingroup\$

I see no problem in your case except the capacity of HDD is limited.

50 % of 1000 Mbps uplink is about 62.5 MB/s to store. A 1 Tb HDD will be filled in full during less than 16000 seconds (about 4 h 30 min).

62.5 MB/s is not guarantied for an HDD as it write speed is about 50-150 MB/s (from here) therefore you need to parallel 2 or 3 of HDDs (e.g. RAID 0). Even low-cost Mini-ATX motherboards integrates two and more SATA interfaces.

Any modern Core-i3 PC could run that task. You can try/test your PC with Wireshark as a software to run the packet capturing.

A LOM Gigabit Ethernet is typically OK, but if you want stronger time stamping you could use specialized capturing hardware like the Endace DAG capturing card.

\$\endgroup\$
0
0
\$\begingroup\$

use double or triple buffering of the data.

remember that a HDD is orders of magnitude slower than RAM. so the buffer sizing in RAM needs to be orders of magnitude large, so there is enough time to perform a HDD write (including seek time) while holding all the new data.

I.E. each buffer needs to be large enough to hold all the input data that will be accumulated while waiting for the HDD to be available for another write operation.

This means the data input should be via a interrupt handler routine that fills a buffer, then passes an indication of a buffer being full and switches to another buffer to continue collecting data.

It helps if the HDD has a very large cache

It helps if the HDD and controller have a through-put that is some multiple of the requisite data rate

it helps if the size of the data buffer being written to the HDD is a multiple of the HDD sector size.

I.E. buffering to RAM will not be the choke point in the proposed application, rather the HDD will be the choke point.

Seek time can be minimized/ eliminated by using a solid state HDD

\$\endgroup\$
1
  • \$\begingroup\$ Thanks very much. I've used data buffering before at the firmware FIFO level, however with the above, are there likely complications due to an OS also using resources? \$\endgroup\$ Commented Mar 24, 2015 at 13:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.