Department of Engineering

IDP: Testing and debugging

Testing

Testing is a significant aspect of any engineering project. In some communities developers write tests first, so that they can serve as a well-defined specification as well as a test-suite. Only once the tests are written do they start writing the code that will pass those tests. Other teams have dedicated testers (computer games sometimes name the testers on the credits). More often, team-members test their own contributions and help test the work of others (read about "Unit testing" and "Integration testing").

Guidelines for testing include

  • Make testing part of the whole process - Produce components that are built with continual testing in mind. Do not assume that once a test is passed, you won't need to perform that test again - a change to the hardware may require a re-test (read about "regression testing")
  • Make testing useful - determine the requirements and then design tests accordingly. A non-specialist might be well placed to list requirements (read about "white-box testing" and "black box testing")
  • Make testing easy - if testing is awkward to perform, it won't be done often. Testing should be easy enough for a non-specialist to do. You could offer a menu that a tester can choose from
  • Make testing results clear - What are you testing? Has the test succeeded? Testing with clear Pass or Fail results are easiest.

Debugging

First, some general points

  • Assume when you're writing your program that it won't be bug-free. Write it so that you (and others) can easily debug it: make routines small and control flow simple.
  • Write defensive code. Always assume the worst. Check return values.
  • Become aware of the mistakes you commonly make - check that you're not going off the end of arrays, that you're using == in the right places, that your using enough brackets and braces, etc.
  • Reduce the scope of your bug search. If you run your program and it seems to do nothing (but doesn't crash or finish) what is it doing? Is it working hard on a calculation, is it waiting for input or is it stuck in a loop? Use "cout" statements to see how far the program gets
  • Debugging becomes laborious if you print out too much diagnostic information and mix it with the usual output. With C++ it might help to
    • use cerr for error messages instead of using cout. By default, cerr output goes to the same place as cout output, but you can separate the streams. For example
               program 2> foo
      
      will run the program sending all the cerr output to a file called foo. Also cerr isn't buffered, so you see the output straight away.
    • have compiler-time control of debug output - You can use the "Conditional Compilation" idea you met earlier. If you enclose the code with
      #ifdef PRINTSENSORVALUE
       ...
      #endif
      
      (what you use instead of PRINTSENSORVALUE is up to you) the enclosed code will only reach the compiler if earlier you have
      #define PRINTSENSORVALUE
      
    • have run-time control of debug output - your program could begin by asking for the amount of debugging information you want (or it could read a value from a file). Your program could then display debugging information in as much detail as the tester requires.

Using a debugger

Programs exist that can be used as a test-cradle for your code. On the Teaching System we have ddd that gives you control over execution of a program, allowing you to step through a program line by line and check the values of variables.

The following program (called average.cc) was meant to print out the mean of 1, 2, and 3.

#include <iostream>
using namespace std;

int average(int num[]) {
  int total;
  for (int i=0;i<=3; i++)
    total=total+num[i];
  cout << "Average is " << total/3 << endl;
}

int main() {
  int n[3]={1, 2, 3};
  average(n);
}

When I ran this program for the first time I got

   Average is -403085996

Staring at the code long enough might help you detect the bug, as might adding some cerr statements. Here I'll show how to use the debugger we have on the Teaching System. Before you can use ddd on a program, you have to compile with a `-g' flag (geany and xcc do this by default). For this example use

 g++ -g -o average average.cc

Then type ddd average. A window will appear showing the source code and some menus.We are going to run the program a line at a time. If you press with the right-button on the word main in your source code, you'll get a menu.

ddd

dddSelect the "Break at mai[n]" option. This will make execution stop when it reaches the main routine. Then click on "Run" in the DDD floating panel. Notice how the arrow moves onto the average(n) line. Hovering the pointer over the n of this line will display n's current value. "Step" again to go into the average routine.

ddd We're interested in the value of total and i. If you click on each of these with the right button and pick the Display option you'll be shown the value of these variables in the top panel. When you continue Stepping through the code you'll see the arrow going around the loop and the variables' values changing. You'll be able to see when total's value goes wrong (i goes from 0 to 3 inclusive, so 4 elements are added together, though only 3 were set).

jjjjjThe commands already introduced may be all you need for the time being if you're using short programs. Watching how the arrow moves may alert you to situations where the control flow is unexpected, and the ability to observe particular variables saves you having to put extra cout commands in your program. Two tips -

  • If you need to provide arguments, use the Run ... menu option
  • If you hover the pointer on an expression like num[i] you'll be shown the value of num (a memory address) or i. To get the value of num[i] highlight it first.

As your programs grow in size, stepwise execution will become more tedious. You need ways of being more selective in your investigations. You can set breakpoints in particular routines, and take bigger steps that don't show what happens in function calls. Browse through the menus to get an idea of the other facilities available. You'll never use most of them, but it's nice to know that they're there. Note that ddd copes happily with multiple source files even if they're in different languages.