Test weighting schemes

[Olly Betts]

Write tests to check that setting the parameters used in the BM25 and traditional weighting schemes works.

[Mikael Johansson]

All test were run against the trec9-train corpus (oshumed.87), english stemmer and the train.batch qrels. I have done extensive testing and genetic algoritm optimization of the BM25 and Cosine constants (haven't checked if I overfit the constants). Xapian and MySQL have the default settings. The fitness column is the recall and hit rank weighted together. Contact me for more stats or other tests. Have a 250 node cluster available, have had thoughts of getting a subset of it to run the genetic tests if it's worth the effort.

Xapian, BM25 Recall Precision Fitness 0.1 0.556 0.125 0.2 0.497 0.244 0.3 0.421 0.384 0.4 0.362 0.508 0.5 0.272 0.696 0.6 0.212 0.853 0.7 0.16 1.006 0.8 0.098 1.202 0.9 0.038 1.48 3-point average (0.2, 0.5, 0.8): 0.289 9-point average: 0.291

Own implementation of BM25 ($K1 = 0.8, $B = 0.5, $C = 0.05, ​http://synd.grow.nu) Recall Precision Fitness 0.1 0.556 0.189 0.2 0.534 0.257 0.3 0.405 0.367 0.4 0.314 0.564 0.5 0.285 0.657 0.6 0.222 0.812 0.7 0.159 1.005 0.8 0.107 1.176 0.9 0.046 1.449 3-point average (0.2, 0.5, 0.8): 0.309 9-point average: 0.292

Cosine measure (low numbers, not quite sure I implemented it correctly) Recall Precision Fitness 0 0.349 0.074 0.1 0.341 0.111 0.2 0.272 0.234 0.3 0.198 0.363 0.4 0.142 0.518 0.5 0.108 0.679 0.6 0.067 0.872 0.7 0.036 1.075 0.8 0.023 1.243 0.9 0.012 1.47 3-point average (0.2, 0.5, 0.8): 0.134 10-point average: 0.155

MySQL Fulltext indexes Recall Precision Fitness 0.1 0.508 0.186 0.2 0.471 0.231 0.3 0.402 0.365 0.4 0.326 0.515 0.5 0.247 0.665 0.6 0.195 0.819 0.7 0.151 0.977 0.8 0.096 1.185 0.9 0.041 1.454 3-point average (0.2, 0.5, 0.8): 0.271 9-point average: 0.271

[Olly Betts]

This bug is about missing feature tests for the Xapian::Weight subclasses, not about evaluation tests of the weighting schemes. Evaluations need doing, but that's not in the scope of this bug!

Fixed in CVS HEAD (apitest now has test weight1).

[Olly Betts]

Fixed in 0.8.4