Arduino Sensor data Logging

Question

I am currently using a Teensy 3.2 to log data from a sensor. My current strategy is to print the sensor data onto the serial port using a 115200 Baud rate. I then use Putty to log the serial data into a CSV file.

The serial port really limits the amount of data samples I can log per second. I want to log as fast as I can. Ideally, I would like to log as fast as I sample the sensor data, but I believe that may be too difficult without an RTOS.

In general, what are the best ways to log data as quickly as possible. I have seen two approaches so far:

1. Use an SD card and write to the SD card. After this, convert that data to a CSV.

2. Store all the data in an array, and then serial.println the array and log that into putty. Writing to the array should be faster, and thus you can collect more data points.

Are there any other ways to go about this? Do you think just writing straight to a CSV as opposed to the serial port would be faster?

Thanks

Answer 1

One way to increase the throughput is to transmit the data in its raw binary format instead of text. For instance, instead of printing the string '3.1415927' (9 characters/bytes), you can simply write the float number as 4 bytes [db,0f,49,40]. This also saves all the CPU cycles that print() would take to convert a number to a string on the Teensy.

However this comes with a few caveats.

• The binary data won't be self-documenting like a CSV file, so you will need to add some instructions on how to read it: how many columns, datatypes, endianness, etc.

• You will have to make sure that the serial connection doesn't automatically convert line endings, as that would corrupt the binary data by silently changing the numbers that happen to correspond to line ending characters.

• If you send whole structs you may need to think about alignment and padding of the fields, or use packed structs.

If you use Python, an easy way to read a binary file with multiple columns is to use np.fromfile together with a structured dtype that describes the binary format for each row.

Writing to an SD card is a valid option, but SD cards have somewhat irregular latency and it can be surprisingly tricky (though not impossible) to interleave writing to the SD card in-between sampling the sensors, unless you have multiple cores or some sort of RTOS.