Data lost writing on Arduino serial port. Overflow?

Question

I have an Arduino connected to a linux box (a Raspberry Pi). I'm reading an analog pin and writing it out with Serial.println() together with the value of millis() The serial port is running at 9600bps

Something in the lines of:

void Setup() {
Serial.begin(9600)
}
void loop() {
    Serial.print(analogRead(A0));
    Serial.print("\t");
    Serial.println(millis());
}

On the receiving end I see a continuous stream of 5 or 6 seconds, then a gap of 2 seconds with no data, then another 5 o 6 seconds full of data. I'm not "seen" it live but plotting the value of millis() with a data read from the serial port.

I think it might be that the data is lost in the write buffer of the arduino, or that something else takes processor time every 5 or 6 seconds (there is an unused ethernet shield attached) or maybe the Serial port writes in bursts.

When I increase the baud rate to 115200 I still get time gaps, but they seem more random.

Update:

Increasing baud rate to 38400 removed the gaps in the samples (gaps measured in arduino millis() time)

Another strange thing was:

On the receiving end I was doing cat /dev/ttyACM0 and piping it to awk which added the system clock to each line received. Then I plotted received time vs system time, and it was not a straight line, but a ladder like plot. Removed the innecesary cat and the time was back to normal.

It turns out cat /dev/ttyACM0 | awk {print} is not the same as </dev/ttyACM0 awk {print}. It seems cat does some strange buffering.

Now I seem to get every value read by the arduino.

Answer 1

analogRead() will return 2- to 3 digit values, say 3. millis() will quickly grow to 4 and 5 character values, and with the tab and the newline, that's about 10 characters generated at each execution of the loop.

At 9600 baud, your serial port can write a bit less than 1000 char/sec or just under 100 10-character lines/sec. Your loop could easily run faster than 100 times/sec and if so the serial output buffer is undoubtedly overflowing. You need to delay 10ms or more in the loop for the serial port to catch up. 115K baud is 12 times as fast so you could reduce the delay (once you know how much you need) by that factor for the higher baud rate.

Answer 2

Two possibilities as I see it:

1. Your program is actually much bigger than what you are showing us and in fact yuo are experiencing some form of stack overflow effectively resulting in random restarts

2. Far more likely, your "problem" is on the Rasp Pi and the Serial traffic incoming is being buffered before being delivered to your (way up the application stack), presumably Python, application

If you want to be sure the bytes get out of the Arduino ASAP, call Serial.flush() at the end of each loop (or after every Serial.print*)