FIFO circular buffer size in over UART

Question

I am trying to implement an UART protocol to make communication between two boards. data are ascii encoded and messages length's are variables. I've chosen to implement a packet in order to "encapsulate" information.

Packet: {STX, DATA, ETX}.

STX is "Start of text" in ASCII table. Its Hex code is 0x02, ETX is "End of text". Its Hex code 0x03. Every byte in DATA field can take any value from 0x00 to 0x7F except 0x02 and 0x03.

Problem 1: Data length
How to send variable data lengths over uart?
Example: {STX, Hello\0, RTX} and {STX, This is a long text\0, RTX}

I've been digging in the internet and I found that the best approach is to use a FIFO circular buffer. The problem is how to fix BUFFER_SIZE in this case? Let's say that I've done some maths and I figured out that the longest message I am going to transmit over UART is 90 Bytes and the shortest one is 4 Bytes. should I #define BUFFER_SIZE 90 or I need to take a larger buffer?

In both cases how does FIFO works in these cases:
Case 1: DATA length < BUFFER_SIZE.
Example: {STX, ABCD, ETX} and BUFFER_SIZE = 90 Bytes.

Case 2: Buffer is full but data are still coming in.
Example: BUFFER_SIZE = 13

Packet: {STX, READ_X, ETX}{STX, REMOVE_Y,ETX}.

In HEX, these messages are
0x02 0x52 0x45 0x41 0x44 0x5f 0x58 0x00 0x03 0X02 0x52 0x45 0x4d 0x4f 0x56 0x45 0x5f 0x59 0x00 0x03

Buffer is full when we reach 0x4d (in Italic). How does it deal with this situation?

Answer 1

Obviously, you need to have some process that is taking data out of the FIFO whenever it discovers that it is not empty. The FIFO then needs to be deep enough to take care of the worst-case mismatch between the putting-in rate and the taking-out rate.

Generally speaking, UARTs are normally slow enough that the other CPU activities can keep up, meaning that the FIFO can be relatively shallow, but if your application is sufficiently complex, you might find that a larger buffer is needed.

For example, if you know that the consuming process can be blocked for some amount of time, the FIFO must be deep enough to hold at least the number of characters that can arrive in that amount of time.

Answer 2

First In First Out. Whatever data comes in, it will be stored into FIFO sequentially starting from a memory location. Suppose the FIFO data width is one byte and the total memory size is 90 bytes, it means it has 90 memory locations and it will store each incoming byte sequentially in the memory locations. You read bytes off it in " First Written First Read" manner. When you read a byte, that location is now "free" to be overwritten. FIFO is said to be full, when you have written bytes to its full capacity but you have not read any byte yet. Suppose the FIFO is full and the data are still coming, what happens is that, new bytes will overwrite the oldest data (from first location) sequentially. Thus data loss occurs. So you have to read the buffer before it gets full. Say 90 bytes are incoming to UART Receiver FIFO at some baudrate. It will get full in a definite time depending on baudrate, before receiving the whole message. If you use FIFO with size < 90 bytes then you have to make sure that you read the bytes before it gets full. If the FIFO size is larger than 90 bytes, and no more bytes are coming in, the data will stay there in FIFO, and you can read those bytes anytime.