firmware testing methods

Question

This is my first question here from the community :)

So I have inherited some bad embedded code, the whole firmware is a messy spaghetti state-machine that has an infinite number of states! and it is so messy that no one dares to touch it because testing it would take months (we don't have any automated testing systems)! so practically code maintenance is almost impossible.

The devices are used in Automation, they are responsible to measure few analogue signals and do calculations to estimate some physical qualities. We have three different types of devices that they even send data to each other and a state of one may impact other devices behaviour.

The hardware is based on SiliconLabs 8051 series. The compiler is Keil uVision with the C51 compiler.

I want to change this sticky situation and organise the code by introducing hardware abstraction layers and modularising the code to smaller chunks.

In order to achieve this goal, I personally think that we need to develop a test bench for devices that I can test them after refactoring and code cleaning. So the idea is to use another microcontroller (STM32) to fake the PCB's or MCU's inputs to simulate the actual environment that the device should work in. Since the behaviour of different devices has an impact on others, I think we have to have a test bench that represents a network of devices so we can also do integration testing.

I have been looking around the web and reading about ways to help me test this messy code, but I haven't found any other way of doing so.

What do you think? is there any other approach that I can take? or do you have any suggestions? how would you handle this situation?

Thanks!

EDIT:

Thank you everyone for your comments and replies!

So The firmware is tightly coupled and there are hardware dependencies and global variables everywhere, the firmware has more than 12,000 lines of code. The main issue here is that even the regression testing involves a hideous and time-consuming 100% manual process that requires the tester to have lots of physical interactions with the device itself and may take more than a day or two!

So I'm looking for a way to test (regression testing for now) the current firmware running on the MCU without the need to refactor and/or changing the code since even a small change in the FW requires intensive manual testing. My understanding is that by having a black-box test rig, that emulates the real environment for the device we can test the FW much quicker and even automate the process. Most of our devices have very limited inputs (like 3 analogue input, and 5 digital I/Os) and the main output is CAN messages that can be processed easily.

Another reason to invest in a HIL test rig is that it will help me to simulate scenarios involving other sub-systems (Integration testing). For instance, there are some ancient bugs in the FW CAN Bus networking that are really hard to identify without monitoring every device and being able to track which one of the devices on the network and during which condition is messing with the network.

Currently, if we want to validify FW we have to install it in an actual environment, collect the data for months ( 2-6 months!!) and analyse the data to validify the device's behaviour. Then you'd find out that there is a bug somewhere (it might even be caused by another device on the network) that is impacting the data you've collected!! Next, you have to do the same process again for another few months :(. I would prefer to automate these process to some extend, so we can at least catch the bugs and issues before installing the device in the real environment.

Answer 1

it is so messy that no one dares to touch it because testing it would take months

Same boat. Don't touch it unless you have to for a hotfix.

I want to change this sticky situation and organise the code by introducing hardware abstraction layers

You will run into the problem that 8051 is not designed to run C, it has very little of everything and the cost of abstraction is too high, especially in a mature product. You probably can't get a chip with more memory since you're at architecture limit.
You're probably also running into a lot of optimization hacks that destroy any last readability the C had.

What you can do however, is extract to inline functions and macros which you can then run in a mockup scenario. You emulate all the requisites for it to compile and then you can test your functions. This is a lot of work to do retroactively.

Some functions you probably can't test due to the side effects. This is a design flaw. You can't "fix" that cheaply.

Tech debt is real, and it's expensive. Don't make a mess again.

---

Note that depending on the compiler 8051 is often used in big-ending mode, this could make testing difficult depending on the micro-optimizations used in the code.

---

On your edit:
Sound like you're still looking to test only the software, but you don't want to take the hardware out of the loop. So you're going to mock the chip environment instead of the chip itself. Result can be equal, just a little bit more work.
Even the act of modifying the code to compile in vanilla gnu C can uncover bugs.

Answer 2

Keil µVision includes a fairly capable simulator for many microcontrollers - even if it tends to crash randomly (it got better the last years). I suggest to read the documentation and start to simulate the chip.

The script language for the simulator can be used to simulate the behavior of external circuitry. You will need to learn about signal functions in that script language.

The script can allocate its own variables. These are completely outside the simulated controller.

The script can also print almost anything you want. You might like to filter the output, though.

The script can be written such that it tests actual behavior against expected behavior.

You can additionally make the simulation and a script run by the command line. This way you can integrate it into any automated development process.

This is a lot of work, but it pays in the end.

I have done this for 8051 controllers, and ARM controllers. Examples of simulated external circuitry are: serial transmission (manchester coded), SD card with FAT file system (connected via SPI, content read from a HEX file), keyboards, ADC.

Last, but not least, the simulator can use DLLs that implement "stuff". You can write your own DLL that simulates any device you want. Example I have done: a memory-mapped display of 5x7 matrix LED display.

Unfortunately all I have done this is closed source, so I can't show anything, just give ideas.

---

Edit:

Some advantages of a pure simulation are:

• The overall behavior is reproducable. This is most valuable if you need to debug or correct a bug.
• The simulated time is most simple to synchronize.
• The simulated time can run faster than in reality, in some cases very much faster. Especially if you need to test long timeouts, this is quite valuable.
• You can "inject" values or behavior that are impossible or nearly impossible to realize with real hardware. Think about a misbehaving device.
• You can observe values really simple that are impossible or nearly impossible to obtain from real hardware. For example, the exact sequence of commands on separate busses.

Some disadvantages should not be ignored:

• You need to understand the simulated peripherals as completely as possible. (But at the end of the day, you need to anyway.)
• It can be a lot to implement.

However, you can replace some parts of the test harness with HiL (hardware in the loop). If you still want to use the simulator, you will need to connect this hardware with the simulator. And I believe that this will need a custom DLL. Note, that the simulation will need to synchronize the simulated parts with the real world, if necessary.

If you want to use just the "testling" and the test rig, you will need to add a lot of sensoring and actoring hardware to automate the tests. There are products that provide such hardware, but you should make estimations on the costs of either way. It will not simplify the task to create the tests. (Note: recommendations are off-topic on SE/EE.)