In my UART communication I need to know the start byte and the stop byte of the message sent. The start byte is easy but the stop byte, not so much. I have implemented two stop bytes at the end of my message, that is \n and \r (10 and 13 decimal). UART only works on bytes 0-255 values so how fail-safe is this? I can imagine, though low probability, that my message might contain the values "10 and 13" after each other when they are not the stop bytes.
Is there a better way to implement this?
There are different ways to prevent this:
20 21 22 23 24 25 10 13
20 21 1b 10 1b 13 25 26
< nr_of_data_bytes > < data >
If your messages are max 256 bytes send:
06 20 21 22 23 24 25
So you know after receiving 6 data bytes that is the end; you don't have to send a 10 13 afterwards. And you can send 10 13 inside a message. If your messages can be longer, you can use 2 bytes for the data size.
Update 1: Another way of defining packets
Another alternative is to send commands which have a specific length and can have many variances, e.g.
10 20 30 (Command 10 which always has 2 data bytes)
11 30 40 50 (Command 11 which always has 3 data bytes)
12 06 10 11 12 13 14 15 (Command 12 + 1 byte for the number of data bytes that follow)
13 01 02 01 02 03 ... (Command 13 + 2 bytes (01 02 for 256 + 2 = 258 data bytes that follow)
14 80 90 10 13 (Command 14 that is followed by an ASCII string ending with 10 13)
Update 2: Bad connection/byte losses
All of the above only work when the UART line is sending bytes correctly. If you want to use more reliable ways of sending, there are also many possibilities. Below are a few:
Note that all above mechanism can be simple or as complicated as you want (or need) to be. In case of resending message, also a mechanism for identifying messages is needed (e.g. adding a sequence number into the package).
How fail-safe is \n\r as stop bytes?
If you send send arbitrary data -> probably not fail-safe enough.
A common solution is to use escaping:
Let's define that the characters 0x02 (STX - frame start) and 0x03 (ETX - frame end) need to be unique within the transmitted data stream. This way the start and the end of a message can be safely detected.
If one of these characters should be send within the message frame, it is replaced by prefixing an escape character (ESC = 0x1b) and adding 0x20 to the original character.
Original character replaced by
0x02 -> 0x1b 0x22
0x03 -> 0x1b 0x23
0x1b -> 0x1b 0x3b
The receiver reverses this process: Anytime he receives an escape character, this character is dropped and the next character is subtracted by 0x20.
This only adds some processing overhead but is 100% reliable (assuming no transmission errors occur, which you could/should verify by additionally implementing a checksum mechanism).
'\x10' DLE (Data Link Escape). Some of the Wikipedia pages suggest that DLE was often used in the opposite way: to say that the next byte was a control character rather than a data byte. In my experience, that's generally the opposite meaning for an escape.
\$\endgroup\$
Commented
Apr 18, 2019 at 21:23
You know, ASCII already has bytes for these functions.
It also has codes for various uses inside the payload.
Your protocol should specify the finest granularity of ACK (0x06) and NAK (0x15), so that negative acknowledged data can be retransmitted. Down to this finest granularity, it is wise to have a length field immediately after any (unescaped) start indicator and (as explained in other answer(s)) it is wise to follow any (unescaped) stop indicator with a CRC.
UART is not fail-safe by its very nature - we are talking about 1960s technology here.
The root of the problem being that UART only syncs once per 10 bits, allowing a lot of gibberish to pass between those sync periods. Unlike for example CAN which samples every individual bit multiple times.
Any double bit error occurring inside the data will corrupt an UART frame and pass undetected. Bit errors in start/stop bits may or may not get detected in the form of overrun errors.
Therefore, no matter if you use raw data or packets, there is always a probability that bit flips caused by EMI result in unexpected data.
There exist numerous ways of "traditional UART quackery" to improve the situation ever so slightly. You can add sync bytes, sync bits, parity, double stop bits. You could add checksums that count the sum of all bytes (and then invert it - because why not) or you could count the number of binary ones as a checksum. All of this is widely used, wildly unscientific and with a high probability of missing errors. But this was what people did from 1960s to 1990s and lots of weird things like these lives on today.
The most professional way to deal with safe transmission over UART is to have a 16 bit CRC checksum at the end of the packet. Everything else isn't very safe and has a high probability of missing errors.
Then on the hardware level you can use differential RS-422/RS-485 to drastically improve ruggedness of the transmission. This is a must for safe transmission over longer distances. TTL level UART should only be used for on-board communication. RS-232 should not be used for any other purpose but backwards compatibility with old stuff.
Overall, the closer to the hardware your error detection mechanism is, the more effective it is. In terms of effectiveness, differential signals add the most, followed by checking for framing/overrun etc errors. CRC16 adds some, and then "traditional UART quackery" adds a little bit.
... I can imagine, though low probability, that my message might contain the values "10 and 13" after each other when they are not the stop bytes.
A situation when a portion of data is equal to terminating sequence should be considered when designing the format of a serial data packet. Another thing to consider is that any character can get corrupted or lost during transmission. A start character, a stop character, a data payload byte, a checksum or CRC byte, a forward error correction byte aren't immune to corruption. The framing mechanism has to be able to detect when a packet has corrupt data.
There a several ways to approach all this.
I'm making the working assumption that packets are framed only with the serial bytes. Handshake lines aren't used for framing. Time delays aren't used for framing.
Send the length of the packet in the beginning, instead of [or in addition to] the terminating character at the end.
pros: Payload is sent in a efficient binary format.
cons: Need to know the packet length at the start of the transmission.
Escape the special characters when sending the payload data. This is already explained in a an earlier answer.
pros: Sender doesn't need to know the length of the packet at the beginning of the transmission.
cons: Slightly less efficient, depending on how many payload bytes need to be escaped.
The payload of the packet is encoded such that it can't contain the start or stop characters. Usually, this is done by sending numbers as their ASCII or Hex-ASCII representation.
pros: Human-readable with common terminal programs. No need for code to handle escaping. No need to know the length of the packet at the start of the transmission
cons: Lower efficiency. For one byte of payload data, several bytes are sent.
"\x0D\x0A". \$\endgroup\$\r\n\r\nwhich contains the\n\rsequence in the middle... \$\endgroup\$