1
\$\begingroup\$

I am currently using a Teensy 3.2 to log data from a sensor. My current strategy is to print the sensor data onto the serial port using a 115200 Baud rate. I then use Putty to log the serial data into a CSV file.

The serial port really limits the amount of data samples I can log per second. I want to log as fast as I can. Ideally, I would like to log as fast as I sample the sensor data, but I believe that may be too difficult without an RTOS.

In general, what are the best ways to log data as quickly as possible. I have seen two approaches so far:

  1. Use an SD card and write to the SD card. After this, convert that data to a CSV.

  2. Store all the data in an array, and then serial.println the array and log that into putty. Writing to the array should be faster, and thus you can collect more data points.

Are there any other ways to go about this? Do you think just writing straight to a CSV as opposed to the serial port would be faster?

Thanks

\$\endgroup\$
5
  • \$\begingroup\$ I want to log as fast as I can. Ideally, I would like to log as fast as I sample the sensor data ... why don't you simply say how fast you would like to log? \$\endgroup\$
    – jsotola
    Commented Jul 2, 2022 at 1:30
  • 1
    \$\begingroup\$ The virtual com port via USB (CDC) is not limited to 115200. You can set 1MBit or more. \$\endgroup\$
    – Jens
    Commented Jul 2, 2022 at 3:00
  • \$\begingroup\$ The teensy has a usb interface that is emulating a com port, it should be capable of data rates well in excess of 115200bps, and in fact the configured data rate probably doesn’t make a difference to the speed (I’m willing to be wrong on that last bit). Have you tried benchmarking it? \$\endgroup\$
    – Bryan
    Commented Jul 2, 2022 at 3:02
  • \$\begingroup\$ @Bryan 12 MBit is physical limit of Teensy 3.2, the other end may have other limits. Yes, the configured baudrate is irrelevant, available USB timing is used. \$\endgroup\$
    – Jens
    Commented Jul 2, 2022 at 3:05
  • \$\begingroup\$ It is difficult to estimate how much data there is per sample and where can you save bytes. Is it 8-bit samples, 10-bit or what? Do you print it in decimal, hex, base64, or what? Is there one sample per line? Do you use one or two bytes for end-of-line marker? \$\endgroup\$
    – Justme
    Commented Jul 2, 2022 at 7:27

1 Answer 1

1
\$\begingroup\$

One way to increase the throughput is to transmit the data in its raw binary format instead of text. For instance, instead of printing the string '3.1415927' (9 characters/bytes), you can simply write the float number as 4 bytes [db,0f,49,40]. This also saves all the CPU cycles that print() would take to convert a number to a string on the Teensy.

However this comes with a few caveats.

  • The binary data won't be self-documenting like a CSV file, so you will need to add some instructions on how to read it: how many columns, datatypes, endianness, etc.

  • You will have to make sure that the serial connection doesn't automatically convert line endings, as that would corrupt the binary data by silently changing the numbers that happen to correspond to line ending characters.

  • If you send whole structs you may need to think about alignment and padding of the fields, or use packed structs.

If you use Python, an easy way to read a binary file with multiple columns is to use np.fromfile together with a structured dtype that describes the binary format for each row.

Writing to an SD card is a valid option, but SD cards have somewhat irregular latency and it can be surprisingly tricky (though not impossible) to interleave writing to the SD card in-between sampling the sensors, unless you have multiple cores or some sort of RTOS.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.