get dynamic length packet from UART?

I Write a program for receive data from Uart ISI, my Data Packet is something like :

2 first byte contain Device ID and type of command. Data is contain some bytes that could be have different length, minimum 1 byte to max 15 bytes, and the last byte represent the packet is finish.

Uart ISI code is:

volatile char UartFlag=Fulse;
volatile unsigned char count=0;
unsigned char coder[13];

ISR     (USART_RX_vect)
{
coder[count]=UDR0;
if (coder[count] == 20)
UartFlag=True;
else
count++;
}


That receive each byte and save in coder. As you see when receive number 20, stop receiving process by UartFlag=True; and in main call a function DoSomthing(); to do something with coder as below:

while (1)
{
if (UartFlag)
{
DoSomthing();
count=0;
UartFlag=Fulse;
}
}


But I have problem that sometime the Data section have 20 in their bytes and its lead to i dont get Correct packet. I try to put \0 (the last byte=0 instead of 20) but maybe have same error.

I am looking for best way to find dynamic length of packet. What is the best way to find the dynamic length of packet?

A way that maybe work is put the data length in first byte of Data section, but it add extra process in sender device.

• Change the finish character to something which won't appear in your data. Can you do that? – Whiskeyjack Jun 30 '17 at 7:18
• no because the packet data maybe contain all of character..and sender dont have limitation in send charachter – Ali Jun 30 '17 at 7:19
• How about this - Keep reading till you encounter stop bit. You don't need to put a special end character on your own. – Whiskeyjack Jun 30 '17 at 7:23
• whats a stop bit? a byte of Data? – Ali Jun 30 '17 at 7:27
• Use multiple characters as an escape sequence, like ### or @#\$%& like many BT or GPRS modules do when switching between command and data mode. – Bence Kaulics Jun 30 '17 at 7:29

In my opinion, to be 100% that it will work in all cases, you have to do what you thought of yourself:

Include the length of the data (the number of bytes you are sending) as an additional header field, for example after the Type field.

This is actually what the Ethernet protocols do.

About extra process on sender I don't know how the sender is implemented. But it doesn't seem to require much processing effort. I mean the sender already knows how many packets will be sent, no? So, as you fill in the other two fields (Device ID and Type), you can fill in this additional field as well.

• thanks.. i think you right ..best way is handle length when packet create..good luck – Ali Jun 30 '17 at 7:43
• @combo_ci - generally you shouldn't be making trivial edits like this, and you especially should not be inserting large blocks of text. Where did that come from??? – Chris Stratton Jun 30 '17 at 14:46
• Beware that simply sending a length is not reliable; you still must have a way to reliably distinguish a "header" encoding the length from any pattern that could occur in data, either by having a multi-byte sequence which is banned from data (possibly broken up by an escape if it occurs), less reliably by a time gap, or perhaps by some additional ("out of band") signalling wire or channel. (Starting the receiver before the transmitter would effectively be out-of-band signalling through the operator's fingers) – Chris Stratton Jun 30 '17 at 14:47
• @ChrisStratton: you are partially right. But I have to say this large block was a comment from me that I subsequently deleted. – nickagian Jun 30 '17 at 15:17
• @ChrisStratton: Unfortunately I don't quite get what you mean :-) The receiver will know that on every transmission the first three bytes coming in are special fields that need special interpretation and not data. From the fourth byte on there will be the data, with a length that was defined from the third special field. – nickagian Jun 30 '17 at 15:19

The two common ways to solve this are the ones you mentioned.

• Use a dedicated length field at the beginning of the package
• Use an unambiguous end marker at the end of the package.

If you don't want to use length fields you have to use an end marker.
If you want to be able to use any of the 256 byte values in payload you have to extend your code by intrducing escape sequences in order to code more than just 256 distinct values.

E.g.

• Use 0x20 0x00 to code original 0x20 in payload (i.e. replace any 0x20 by 0x20 0x00 in payload)
• Use 0x20 0x01 as end marker.

Since escaping increases the size of the coded payload it is advisable to use an escape character value that is unlikely to appear in original payload data (so 0x20 wouldn't be a good choice if payload is ordinary ASCII text).

• Note that a length fields requires reliable detection of a header vs. arbitrary patterns in data. This isn't necessarily any simpler than detecting an end marker. – Chris Stratton Jun 30 '17 at 14:49
• Common protocols that use the escape character technique are SLIP and Asynchronous PPP. – Kevin White Jun 30 '17 at 19:34
• @Curd: Unfortunately, I do not know at all what data from the sender of the oven, maybe 0x20! ( because the sender mix some bits together and each bit means an action...mixing the bits in some condition may deal to create 0x20 and its a big bug in our system – Ali Jul 1 '17 at 4:33

Another thing to consider is how the data field is encoded. You can avoid a length specifier in the packet by ensuring that the end marker is not ever expected in the data field. (Note that in applications that I create I also use a start marker that is unique and cannot occur in the data region. This allows quicker re-sync to the command stream without losing one extra packet). So if a data region is just numbers for example encode them in character format as opposed to binary format. Then also select your device ID and packet Type as a readable character. This scheme can leave you a series of bytes to be used as "control characters" two of which you use as the start marker and end marker. The ASCII character set neatly reserves the encodings of 0x00 to 0x1F as control characters. This is MANY years old and so you can see that this problem was solved long ago. There are even names given to some of them that can be directly leveraged such as SOH (start of header) and EOT (end of text).

• you say:_Note that in applications that I create I also use a start marker that is unique and cannot occur in the data region_, my big problem is that every character maybe occur in data section,character or Number – Ali Jul 1 '17 at 4:17
• @MichealKaras could you pealse explain more about This allows quicker re-sync to the command stream without losing one extra packet? – Ali Jul 1 '17 at 4:21
• @combo_ci - Yes! If you have a receiver is connected and starts looking for packets to arrive and is not yet in sync between its incoming data stream and the parsing of active packet boundaries the quickest way to achieve sync and not lose a packet is to look for a unique start marker. If you look for an end marker instead you cannot trust most of what came before so you must discard what may have been almost all of a packet and concentrate on receiving the next packet. – Michael Karas Jul 1 '17 at 17:02
• A system timer interrupt is more applicable to working out multi byte gap between packets. Back to back bytes is better handled by reading a continuously running counter each time you read the next UART byte to get a relative time stamp each byte as received. Such counter wants to be ticking at a rate at least the bit rate of the UART stream but have enough bits that it does not over flow at more than say the length of time of several packets. – Michael Karas Jul 1 '17 at 17:56
• @combo_ci - I do have to comment that you say that you are able to change the transmit protocol (ie adding a modified device ID) and yet you complain that you cannot do anything about the data content. I have to suggest that if you have control of the transmit end then you can take steps to solve the problem that you have!! One brute force way is to resort to the ASCII printable characters solution and convert all the arbitrary source data onto a Hex ASCII format by encoding each source binary byte to a readable character format in two bytes. Convert it back to binary at the receiving end. – Michael Karas Jul 1 '17 at 18:04