15
\$\begingroup\$

Every textbook I've seen makes a big deal of the fact that testing and verification are two different concepts. Yet none of them provides a clear (or clear enough to me, at last) distinction.

To provide some context, I'm interested in the verification of digital hardware designs using hardware design languages (HDLs).

I've seen some explanations that resort to a "physical" or "tangible" difference: if it is about a manufactured device, then it's testing. Is this the whole story? If so, why does the word "test" come up so often in verification (especially in functional verification, we talk about testcases, testbenches, DUT (device under test), directed tests, random tests, etc...)

\$\endgroup\$
21
\$\begingroup\$

Was an ASIC Design Verification Engineer at Qualcomm. In the most simple way I can explain it:

Testing: Making sure a product works, after you've created the product (think QA).

Verification: Making sure a product works BEFORE you've created it.

They're both testing, just that verification is more complicated because you have to figure out a way to test the product before it exists and you have to be able to make sure it works as designed and to spec when it actually comes out.

For example, Intel is designing their next processor, they have the specs, they have the schematics and the simulations. They spend $1 Billion USD to go through fabrication and manufacturing. Then the chip comes back and they test it and find out that it doesn't work. They just threw a lot of money out the window.

Throw verification in. Verification engineers create models that simulate the behaviour of the chip, they create the testbench that will test those particular models. They get the results of these models and then they compare it with the RTL (model of the circuit writting in a hardware design language) results. If they match, things are (usually) OK.

There are a number of different methodologies for the verification process, a popular one is Universal Verification Methodology (UVM).

There is a lot of depth in the field and people can spend their entire career in it.

Another random tidbit of information: Usually you need 3 verification engineers for 1 design engineer. That's what everyone in the field says anyway.

EDIT: A lot of people think of verification as a testing role, but it's not; it's a design role in itself because you have to understand all the intricacies of your IC like a designer does, and then you have to know how to design models, testbenches, and all the test cases that will cover all the feature functionality of your IC, as well as trying to hit every single line of RTL code for all possible bit combinations. Remember that a processor nowadays has billions of transistors due to the fabrication process allowing smaller and smaller (now 14nm).

Also, in large corporations like Intel, AMD, Qualcomm, etc, designers don't actually design the chip. Usually the architect will define all the specs, layout the types of pieces that need to go together to get a particular function with a specific requirement (i.e. speed, resolution, etc.), and then the designer will code that into RTL. It's by no means an easy job, it's just not as much designing as a lot of engineers coming out of school think it is. What everyone wants to be is an architect, but it takes a lot of education and experience to get to that point. A lot of architects have PhD's, and like 15-20 years of experience in the field as a designer. These are brilliant people (and sometimes crazy) who deserve to be doing what they're doing, and they're good at it. The architect on the very first chip I worked on was a bit awkward and didn't really follow some social norms, but he could solve anything you're stuck with regarding the chip, and sometimes he would solve it in his head and tell you to look at one signal and you'd be like, "how the hell did he do that?". Then you ask him to explain and he does and it goes way over your head. Actually inspired me to read textbooks even though I've graduated already.

\$\endgroup\$
  • \$\begingroup\$ +1 Thanks for the last remark, helps us see that the field is indeed important (though RTL and design engineering sounds more attractive to most engineers, I think) \$\endgroup\$ – VHDL Addict Nov 6 '14 at 23:05
  • \$\begingroup\$ For completeness, would you mind adding what a testcase is? \$\endgroup\$ – VHDL Addict Nov 6 '14 at 23:07
  • \$\begingroup\$ I added a tidbit about what verification actually is because of your first comment about the design role being more attractive; they're both good roles, just depends on what you like. As for a test case, a SoC like the Snapdragon could have tens to hundreds of thousands of test cases, and with random testing, in the millions. To put it simply: you're applying a set of input bits and that gets changed going through many modules, then you get some output bits as a result that you compare with your model's results. Something as simple as testing a picture showing up on your phone would have... \$\endgroup\$ – PGT Nov 7 '14 at 3:31
  • \$\begingroup\$ a large number of testcases. Say you want to show a single pixel on your mobile screen. What someone outside the field will think is apply 1 bit for white, and 0 for black. In the actual mobile world, that pixel can differ by size, intensity, rotation, color format (YUV###,RGB###, etc.). And you're probably testing 1 bit within a set of bits that gets applied to the input together. The other bits may be 0 because it's black, or may be 1 because it's handling other information like how to handle transmission mode, CLK, enables/disables, triggers, fancy stuff like that. \$\endgroup\$ – PGT Nov 7 '14 at 3:42
6
\$\begingroup\$

In my book Verification is ensuring that what you have designed "does the job" - i.e., you have a set of things the "device" needs to perform, and verification ticks those off on the list.

Testing, though, is making sure that the things the "device" does are done right. You have a set of functions, and you test each function making sure that function performs properly.

In a nutshell, Verification is checking the design, and Testing is checking the product.

\$\endgroup\$
  • \$\begingroup\$ I think I'm starting to understand... Could you please provide some examples of each? \$\endgroup\$ – VHDL Addict Nov 6 '14 at 21:56
  • \$\begingroup\$ How does that fit with a verification plan, which specifies what should be implemented and also how to know that the functionality is correct? It would be of little use to implement or tick-off a function if it does not work. \$\endgroup\$ – VHDL Addict Nov 6 '14 at 21:58
  • \$\begingroup\$ @ Majenko - So you have written a book on Verification? Would you share some more details about it? \$\endgroup\$ – Michael Karas Nov 7 '14 at 2:53
4
\$\begingroup\$

Coming from a an ASIC (hardware) design background, there are three important terms: validation, verification, and test. The earlier answers generally talk about one or two of these terms, but don't clearly contrast all three in the way I would. Here's the way I understand them:

  • Validation: does the specification (often a C model) meet market or customer requirements
  • Verification: does the implementation (RTL, netlist, or GDS2) match the specification
  • Test: does the manufactured device match the implementation
\$\endgroup\$
  • \$\begingroup\$ Can the netlist and GDS2 simulations give different results? \$\endgroup\$ – Ciro Santilli 新疆改造中心996ICU六四事件 Jul 15 '16 at 18:03
  • 1
    \$\begingroup\$ @CiroSantilli巴拿馬文件六四事件法轮功, I suppose you're asking about behavior of gates vs. transistors. For normal digital voltages and waveforms, I'd say they will give same results. But there could be "analog" effects not considered by idealized gates, such as power/ground variations or signal charge sharing. If those effects are present, then ideal digital behavior may not hold true. \$\endgroup\$ – Winston Smith Jul 30 '16 at 23:31
1
\$\begingroup\$

A test is designed to see if a specification is met. Verification is to see if the device meets the design inputs -- i.e., all the specs. I suppose there are many more interpretations, but this is what I've seen in FIA guidance docs.

\$\endgroup\$
  • \$\begingroup\$ I see, I've changed the wording a little (from test to testing) to make it clearer that both are processes. I agree with you that for individual tests the word test is appropriate (sometimes I feel like I'm restating the obvious with these terminology questions...) :) \$\endgroup\$ – VHDL Addict Nov 6 '14 at 21:22
1
\$\begingroup\$

We make a distinction between verification testing and validation testing. Let's say you are designing a fan that cools off some equipment. Verification testing is done to make sure the fan meets all the design requirements. So you might test airflow, thermal cycling, vibration, etc.

Validation testing makes sure the design requirements were the right one. Did the design inputs we had for the fan actually give us the fan we wanted? For example you'd make sure the fan actually cools the equipment as intened.

\$\endgroup\$
  • \$\begingroup\$ This is how the terms are understood by the books on Software Engineering I have read. Validation = make sure the requirements are right (check them with the customer, regulations, etc.); Verification = make sure the product is right (test it against the specification) \$\endgroup\$ – Wouter van Ooijen Nov 7 '14 at 7:40
1
\$\begingroup\$

ISO9000 talks about verification and validation. In the context of ISO9000 verification means testing a prototype design to prove it meets functional and performance expectations. Validation means testing the first production run also meets design expectations. Verify first, validate later is my little way of remembeeibg the order of things.

Several software standards reverse the order of verify and validate and this could really cause confusion so be aware of this.

Bottom line is... Why are you testing something - is it a prototype design - if so then the quality standards tend to call this verification. If you are testing a production run for the first time then hardware guys call this validation.

Just my personal experiences.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.