Testing and debugging

Project Discussion Foundations Resources
1

Python

Use pyflakes to check for common errors. Hook this up to your preferred test harness using something akin to the following:

Mock is one of the better mocking libraries and is already used in ubiquity, software-center, and apport.

ipdb iPython-like debugging shell with superior tab completion is pure heaven:

import ipdb; ipdb.set_trace()

C

Tracing

  • strace: system call tracing
  • ltrace: library function tracing

Memory Checkers

Note: to use memory checkers reliably, you need to tell (e(g))libc and glib to disable all their checks too, like this:

export G_SLICE=always-malloc
export G_DEBUG=gc-friendly,resident-modules
export MALLOC_CHECK_=0

Note the trailing underscore at the end of the last variable!

valgrind

No need to recompile code.

Basic memory checking:

valgrind -v myprogram --arg=foo -baz

More aggressive checking:

valgrind -v \
  --track-fds=yes \
  --log-file=/tmp/valgrind.log \
  --tool=memcheck \
  --leak-check=full \
  --track-origins=yes \
  --malloc-fill=0x0 \
  --free-fill=0x0 \
  --show-reachable=yes \
  myprogram --arg=foo -baz

Deep magic, which will spawn gdb when an error is detected (--db-attach=yes):

valgrind \
  -v \
  --trace-children=yes \
  --track-fds=yes \
  --time-stamp=yes \
  --db-attach=yes \
  --read-var-info=yes \
  --tool=memcheck \
  --leak-check=full \
  --leak-resolution=high \
  --num-callers=40 \
  --show-reachable=yes \
  --track-origins=yes \
  --undef-value-errors=yes \
  --freelist-vol=60000000 \
  --malloc-fill=0x7 \
  --free-fill=0x8 \
  myprogram --arg=foo -baz

dmalloc

Install:

sudo apt-get install -y libdmalloc5 libdmalloc-dev

eval `dmalloc -b -i 1 -l /tmp/dmalloc.log all`
myapp --arg=foo -bar
cat /tmp/dmalloc.log

Using dmalloc with LD_PRELOAD:

(eval `dmalloc -b -i 1 -l /tmp/dmalloc.log all`; LD_PRELOAD=libmalloc.so.5; ./myapp --arg=foo -bar)

Static Analysis

cppcheck

check:
    ...
    if type cppcheck >/dev/null 2>&1; then \
        cppcheck -v . --error-exitcode=1; \
    fi

splint

Excellent static analysis tool. May now be unmaintained? Does not understand C99 (variadic macros).

splint -I/usr/include -I/some/where -DMY_DEFINE=1 -DFOO src/main.c 2>&1|tee splint.log

LLVM/Clang:

sudo apt-get install -y clang
scan-build -v -v -v make 2>&1|tee scan-build.log

smatch

Smatch is a static analysis that uses Linus sparse C parser.

make clean
make CHECK="smatch --full-path" CC="cgcc -std=gnu99"

Performance testing

judge runs two commands 50 times and provides the following report:

ubuntu@server-8149:~$ ./judge/judge ./2281 ./with_changes
^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^v^

    n mean sd min max cmd
   50 10316.2ms 355.7 9945.9 11638.6 ./2281
   50 9867.5ms 625.8 9385.6 13061.3 ./with_changes
 -448.636ms -4.3% p=0.000
difference is significant at 95.0% confidence (p=0.000):
based on these samples, suggested sample size is n>=283 to have a 112.16ms confidence interval

Cloud infrastructure

cloud-init provides a means to run scripts during the creation of a Canonicloud instance. You can use this to create configurations for the infrastructure you need for a project and rapidly deploy that, either to develop or run tests against.