Acquire Testing Savvy

By Richard A. Quinnell

As our systems become more capable and complex, their testing becomes more complex as well. In fact, as the networking special feature in this issue points out, testing complexity increases faster than the complexity of the systems. Because rigorous testing requires examining all possible combinations of system state, input variables, and in some cases environment, linear scaling of system capability often results in exponential scaling of test requirements.

Unfortunately, some design engineers view rigorous test as a bitter medicine, something to be avoided as long as possible and gotten over with as quickly as possible. They often rationalize the avoidance by telling themselves that testing is a relatively simple, if tedious, task that does not merit much attention or planning. Both concepts are flawed.

Numerous studies have shown that early and frequent small tests followed by more intensive regression testing will speed product development and avoid major problems during final system integration and shakedown. Further, by considering test needs up front and planning for the access and modes that production testing will require, product development will result in faster production ramps and lower manufacturing costs. Some of these concepts are illustrated in the technology applications article on unit testing in this issue.

Take the Test Test

The error in the idea that test is relatively simple is difficult to demonstrate through studies; it is best seen through experience. Early in my engineering career an insight into the complexity of rigorous testing came from an instructional example I ran across in the literature. The example challenged readers to create a set of test cases for a program that read three numbers off of a punched card (I told you it was early in my career) and evaluated them. The program would interpret the numbers as the lengths of the sides (A, B, C) of a triangle, then determine if the described triangle was acute, obtuse, isosceles, equilateral, or a right triangle. A blank card terminates the program.

It seems a simple enough program to test. We all know to supply an example of each triangle type along with a blank card and see that the program correctly identifies them, a total of six test cases. The place many fail, however, is in evaluating how the program responds to errors. Test of this simple program should also include:

  • Cards with only one or two numbers in differing positions (6)
  • A card with four numbers
  • Cards containing zeros in each of the three positions (3)
  • Cards containing zeros in any two of the three positions (3)
  • A card with zeros in all three positions
  • Cards containing letters instead of numbers in any or all positions (7)
  • Cards with numbers that do not represent a complete triangle, such as those with A+B<C (open) and A+B=C (a line)

There may be more test cases needed that I have since forgotten, but even these represent nearly 25 additional test cases beyond the obvious six. Clearly, testing is more difficult than it first appears and needs to be done thoroughly in order to avoid unexpected results after system deployment.

Yet, test is a two-edged sword. While it is clear that early and rigorous testing during the design phase can save time and money, excessive, inefficient, or unnecessary testing can burn though time and money even more quickly. The prime example for the cost of testing was the U.S. Military procurements during the ‘70s. Standards and specifications called for so many tests during and after manufacture that the legendary thousand-dollar hammer was not much of an exaggeration; testing added major expense to military system costs.

Finding the degree of testing that minimizes total system development and production cost is not an easy task Fortunately, designers do not need to identify that ideal test configuration; that’s the job of the test or manufacturing engineers. What designers must do, however, is keep test needs in mind throughout the design cycle. Maintaining a dialog with the test and manufacturing engineers is a good place to start. You do not need to become test experts, but you do need to become test savvy.