How to deal with start sequence in payload of communication packet?

Basic serial messages are being transferred between an embedded device and a PC app.

MCU UART Peripheral -> FTDI cable -> PC USB Serial Library


The messages are currently fixed-length (32B) and being transferred in the format:

start_low_byte start_high_byte payload_0 payload_1 ... payload_(n-1) end_low_byte end_high_byte


Where not all of the payload bytes are necessarily being used.

Now I know there are many tradeoffs that are application dependent (fault-tolerant, bandwidth => overhead/payload ratio limitations, range of the values in the payload, etc). In this case, I'm trying to transfer an arbitrary payload which makes me wonder:

If needed, how can I ever transmit a start or end sequence? And is there any advantage to having a separate start and stop sequence, or is a single delimiter sequence OK?

If I escape the start/stop sequence, then my message seems as though it can grow up to twice as long.

Firstly I don't usually have an end marker. I would normally have a payload length as part of the header.

Secondly I would usually incorporate a checksum, usually at the end of the payload (where you have the end sequence).

As for escaping characters used in the header when they appear in the payload: Yes. How you escape it is up to you, it's your protocol. Options include:

1. Using a pre-defined "escape" character (27 is commonly used) which itself needs escaping.
2. Double up any reserved header characters. So say the header is 0x55 0xAA then you would transmit that in the body as 0x55 0x55 0xAA 0xAA and then recognise the double characters as a single one when decoding it.
3. Use different data sizes for header and payload. If you restrict the payload to 7 bits per byte then have the header characters with the high bit set they can never appear in the body. Makes the body bigger though.

If you're looking for a specific sequence for the header then it doesn't matter if those header characters appear individually in the body, only the entire sequence.

• Is that common escape character 27 decimal (0x1B)? Or 0x27? – tarabyte Feb 6 '15 at 17:48
• 27 decimal - also known as ESC ;) – Majenko Feb 6 '15 at 17:49

You can use a single deliminator sequence but it may cause your code to time out. You can escape the sequences but size growth is expected. Instead you can include a frame size field in the header so no escaping is required.

You can take my Simple Serial Transfer Protocol as used in my various projects as an example. It is simple enough to understand, but it also provides an IPv4 compatible mode and multi-drop with CAN PHY.

Here is the structure of the SSTP frame described using C-like pseudo code, sent in big-endian format (aka network byte order)

struct sstp_frame {
uint16_t magic;  // = 0x55aa
uint16_t proto;  // For simple UART communication send 0, in IP-compatible mode
// send IP next protocol number
uint16_t size;   // frame length, includes header, and footer

uint8_t  body[]; // body of the frame, unmodified

// frame footer
uint16_t crc;    // CRC of the body of the frame
uint16_t magic2; // = 0xaa55
}


After every frame is sent the line is kept silent for the same time it takes to send the frame.

The receiving end waits for the header magic, reads the size of frame after the header magic, calculates the CRC while receiving the packet and compares it, and checks the ending magic. Any mismatch (bad CRC or wrong magic) or timeout cause the packet to be discarded.

If CAN PHY or any other mean is used to achieve multi-drop, speak only when the line is idle. Read back every byte after sent, and whenever a mismatch happened abort the packet and wait for the same length as the packet is sent so far if header is sent, or a pseudo-random time if header collision is detected.

If you need reliability, you can run TCP on top of this protocol, using its IPv4 compatibility mode.