Automated coverage testing

[Olly Betts]

We should have automated coverage testing, to help use improve the coverage of our testsuite.

Suggested approach:

• Use buildbot to automatically generate a suitably instrumented build from SVN with the right options: -fprofile-arcs -ftest-coverage and probably -O0

• Use ​lcov or something similar to run the testsuite under gcov and generate HTML of the results. Probably running this as part of the buildbot job is easiest.

• We ideally want to keep a record of some appropriate "summary statistics" so we can graph how we're doing over time. Hopefully there is a suitable utility which will produce suitable stats and we just need to note these for each run along with the SVN rev and branch URL. Failing that, perhaps we can parse them from the HTML output or something.

Assigning to Richard for now, as he's best able to do the buildbot part at least.

[Olly Betts]

OK, I gave this a go on atreus, and it failed:

mkdir xapian-coverage
cd xapian-coverage
svn co file:///home/xapian-svn/xapian/trunk xapian
cd xapian
./bootstrap
cd xapian-core
./configure CXXFLAGS='-O0 -fprofile-arcs -ftest-coverage' --enable-quiet --enable-maintainer-mode --disable-visibility
make

I used --disable-visibility hoping it would fix the error I got, but it doesn't help. The relevant part of the output is:

/bin/sh ./libtool --quiet --tag=CXX   --mode=link g++ -Wall -W -Wredundant-decls -Wpointer-arith -Wcast-qual -Wcast-align -Wno-long-long -Wformat-security -fno-gnu-keywords -Wundef -Wshadow -Werror -O0 -fprofile-arcs -ftest-coverage   -o bin/xapian-check bin/bin_xapian_check-xapian-check.o bin/bin_xapian_check-xapian-check-flint.o  libchertcheck.la libflintcheck.la libxapian.la 
/usr/bin/ld: bin/.libs/xapian-check: hidden symbol `atexit' in /usr/lib/libc_nonshared.a(atexit.oS) is referenced by DSO
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: ld returned 1 exit status
make[2]: *** [bin/xapian-check] Error 1

Google finds a thread with this error in, but it seems unrelated to what we're doing here...

I also tried with -O1 and that doesn't seem to help.

[Olly Betts]

OK, just found that doing a static build fixes this:

./configure CXXFLAGS='-O0 -fprofile-arcs -ftest-coverage' --enable-quiet --enable-maintainer-mode --disable-shared

[Olly Betts]

I've now added this to buildbot.