UART not fast enough for debug purpose, what alternatives exist?

Question

This problem is related with debugging an FPGA design implemented in Intel/Altera FPGA.

A custom debug mechanism has been introduced into the design. This new part of the design writes into a FIFO when certain events occur during design operation. The data exchanged and the events that caused this exchange are stored in this FIFO. The FIFO is then read by a Nios II. After some basic processing (check what is contained in the data), the Nios prints out a trace to a JTAG UART. This trace describes what events are occuring as the design operates and the value of the data moving from one part of the design to another. A Nios terminal runs on the computer to print the messages on the screen.

Using signal tap, system console or any other method is not possible as the events in the system and the data moving from one place to another must be presented in a human readable form over several seconds of design operation.

Now here is the problem. It seems that the calls to printf are too slow. The FIFO soon overflows and when this happens, information about many events that occured does not get written into the FIFO. The data rate into the FIFO cannot be predicted as it is not filled at a constant rate. Increasing the FIFO depth or increasing the depth of the buffer in the JTAG UART is not sufficient. Using alt_printf or some other form of optimization also does not fix the problem.

Since printf with JTAG UART is too slow, what alternatives exist? Since this is not a development board but a custom PCB, I do not have many pins available to use for debug purpose. There are many possibilities of course.

EDIT:

The trace looks like this:

START
MSG 0 DL
  MSG 0 UL
  DATA1 3200 DATA2 350
MSG 1 DL 
  DATA1 500
  NO REPLY
MODE2
MSG 0 DL
  MSG 3 UL
  DATA1 400 DATA2 100
MSG 10 DL
  DATA 0x100
  NO REPLY
MODE1
...

Please note that the above is a very oversimplified view but you get the idea.

Answer 1

If I understand correctly, data flow is like:

• Binary data inserted into FIFO

• Nios reads binary data and formats it using printf()

• This text data is sent to PC via JTAG UART

First thing you need to do is figure out if the bottleneck is the JTAG UART (these tend to be very slow) or the printf() formatting on Nios. You can do this simply by making Nios run a "while(1) printf("hello\n");"

Then look at the data rate received by the PC. This should run your virtual JTAG UART at its maximum speed, so you should get lots of data. If it is still very slow, then suspect the JTAG UART. If it runs over USB, then remember USB interrupt transfers only run every frame, so a dumb implementation which flushes every character would max out at one character per USB frame... Check if you can tweak its settings. Is it flushed after every line? Maybe you can use a larger buffer. Search the forums for people having slow UART issues.

If the JTAG UART is indeed the problem, perhaps try a real UART, which only needs one wire for debug messages, and a USB-serial dongle. Some of those can reach pretty high bit rates.

Then you can independently measure printf() speed by printing to a buffer, and NOT sending it to the uart. Just use sprintf() and make sure the optimizer does not remove this "useless" code. You can measure CPU load on your Nios: when the CPU is idling waiting for FIFO data, set a pin LOW. When it is busy processing data, set that pin HIGH. Then probe the pin with a scope, and you'll see cpu load as duty cycle. Likewise you can measure accurately how long a sprintf() takes by toggling GPIO before and after, and probing it with a scope.

If printf() is the problem, you can use a faster implementation, or you could just send FIFO contents to the PC, and do the formatting on the PC.