I'm testing an electronics toy product. Any advice on writing test plans and test scripts.

I'm thinking along the lines of listing user actions:

  • for normal input/response
  • that are suppose to cause exceptions/errors

What else am I missing?

We have the flow chart that our vendor is using to program. I think I'm going to look at the flow and test each decision point to make sure that all the YES and NO paths are correct.

Any other things I need to include? Any other strategies that would make this more efficient? Should I write a test script that can check off a few different decision points or do I need to test each in isolation? (I don't know if this is even possible because to get to some s/w decision points I need to get pass others.

  • 1
    \$\begingroup\$ Test anything you care about being correct in the field. Note that firmware only needs to be qualified once, but hardware-related failures need to be tested every unit. \$\endgroup\$ Mar 27 '12 at 17:32

Prior to starting designing and during design you should have created a number of documents, like User Requirements Specification (URS) and Functional Requirements Specification (FRS).

Everything you specify sooner or later has to be tested!

Why else would you take the trouble to put it on paper?
The URS should be where you derive the User Manual from, but also the User Test Specification. Describe what you expect to happen when the user performs an action, like pressing a button, under which circumstances. Pressing button 1 may cause a different action depending on the state of switch A.
So you describe start conditions, actions, expected end conditions. For all normal user actions.

Then there's unintended use. Organize a creativity session to discover as many ways as possible to abuse the product. If you know how to conduct a brainstorm you could use that. Issues could go from pressing several buttons simultaneously to inserting batteries the wrong way. Can you insert the wrong type of batteries? Again you describe start conditions, actions, expected end conditions. If you don't know what the result of an (even unintended) action should be then you missed something during design.

This implies that writing a Test Specification (especially the part about unintended use) shouldn't be done just before testing. The software designer must know what should happen if you press two buttons at the same time before she starts writing the software. Remember that the cost of changes in a design rises exponentially with time.

It is of course that device failure under certain conditions is allowed. You may not have protection against inserting batteries the wrong way and you know the device will be damaged if the user does. (Not a good idea IMO, just an example.) Also then this has to be described in a functional spec, to show that you haven't overlooked it. I would even mention it in the test spec with the comment "test not required".

It should be obvious that pressing two buttons simultaneously shouldn't lock up the software. But don't say it's impossible. The software designer must be able to show in her code how she deals with this. Which is even better than just trying it. During testing one button may be pressed slightly later than the other.

  • \$\begingroup\$ Great points @stevenvh. One case that you list that I didn't think of is inserting the batteries the wrong way. Hopefully the engineers that taken care of this, but we need to test this to be comprehensive. \$\endgroup\$
    – milesmeow
    Mar 27 '12 at 19:12
  • \$\begingroup\$ Measures against inserting a battery the wrong way belong in the design and implementation phase, not in the test phase! (verification of these measures of course does belong in the test phase) \$\endgroup\$ Mar 27 '12 at 21:07
  • \$\begingroup\$ I find funny that you refer to the software engineer as to a woman...did you have a bad experience? :) \$\endgroup\$
    – clabacchio
    Mar 28 '12 at 7:28
  • \$\begingroup\$ @clabacchio - Just trying to break with tradition. And I don't say anything bad about her, do I? (For what it's worth, I'm not religious, but I also refer to God as "she" :-)) \$\endgroup\$
    – stevenvh
    Mar 28 '12 at 7:33
  • \$\begingroup\$ No, no, nothing negative...probably it's just because I'm used to refer to engineers as males, because in my language it's a masculine word :) \$\endgroup\$
    – clabacchio
    Mar 28 '12 at 7:39

The best way to test the firmware and hardware design is to connect the whole circuit up to something like an Arduino. The Arduino would then apply various inputs to the toy, and measure the outputs. This can even be used for per-part tests.

If I have enough FLASH, I often write firmware tests in the firmware. In some cases, it's possible to get the firmware to do some tests on the hardware too. For example, it might test that the microcontroller pins are all working by trying to pulse them high and low very briefly, and testing if they did that.

Doing this creates a much more thorough test of the hardware, and the good thing is that it costs nothing on a per-part basis. (except possibly for the extra FLASH required)

The firmware would also contain some tests of basic functions that are easy to get wrong: hand written signed 32-bit multiply on an 8-bit MCU for example.

It would also contain compile time assertions to check that the integers are the correct size.

Doing all this has frequently caught bugs immediately, and saved me many days of debugging. It can also catch faulty or ESD damaged hardware.

  • \$\begingroup\$ I like this automated method. I think it would pair nicely with physical testing...with a real person too. Especially consumer products...you may find out something about the interaction that may need to be tweaked. \$\endgroup\$
    – milesmeow
    Mar 28 '12 at 19:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.