Merge learning to rank branch

[Richard Boulton]

We should work on merging the letor branch to trunk, to ensure it doesn't bitrot and get lost.

To get an overview of changes on the letor branch, see ​https://github.com/xapian/xapian/compare/master...svn%2Fgsoc2011-parth

Several things need to be done. Initially, by parth:

• Done: add the letor/dir_contents file, so the build system works.

• Done: remove letor/svm.h; the installed svm.h should be used instead.

• Done: add the formulae to the docs/letor.rst file (in latex form, we can worry about making them display nicely later).

• Done: add a description of which versions of libsvm should work to the docs/letor.rst file.

• Done: remove the include of letor.h from xapian.h, and add a comment to the top of letor.h to indicate that it is not part of the official stable API.

At this point, parth can submit his code to google, and the gsoc project is finished.

Then, by olly or me:

• Done: merge recent changes in trunk to the branch

• Done: merge recent changes in trunk to the branch

• Done: add detection of libsvm.h to configure.ac

• not relevant now xapian-letor is a separate module: make the build skip letor if svm.h isn't available

• make the code conform to the coding conventions we use in the rest of the code (largely done)

• add automated tests (parth will try to do this)

• Done: anything else required to merge the branch to trunk

• check that at least one buildbot is building the letor code.

I've marked this as milestone 1.3.0 so it doesn't get forgotten, but consider that a soft milestone.

[James Aylett]

What's the minimum version of libsvm required? 2.91 is in Ubuntu 11.04, but it's not the latest by any means. (Even assuming it's the same libsvm.) 2.85 is in Debian oldstable, and I've put it on atreus, but I suspect that's way out of date…

[parth]

Replying to james:

What's the minimum version of libsvm required? 2.91 is in Ubuntu 11.04, but it's not the latest by any means. (Even assuming it's the same libsvm.) 2.85 is in Debian oldstable, and I've put it on atreus, but I suspect that's way out of date…

libsvm-3.1 was used during the development of the Letor Project. By the way, my system works fine with system installed libsvm-3.0-2 too.

Parth.

[Richard Boulton]

The version of libsvm to use should be noted in the docs/letor.rst file.

[Olly Betts]

Currently the 1.3.0 milestone is really "stuff to address before 1.4.0" so I've created 1.3.x for that role leaving 1.3.0 for stuff to really do for 1.3.0.

[Olly Betts]

Pencilling in for 1.3.2.

[Olly Betts]

I've updated the description to mark what has been done, and to note a few more things to do.

I wonder if it would make more sense to make this a separate top-level module (e.g. xapian-letor), rather than an optionally built part of xapian-core?

[Olly Betts]

Parth was in favour of a separate xapian-letor module (either on IRC or comments in melange).

It also avoids having to make libsvm a requirement for all users, or having an --enable-letor configure option and forcing packagers choose whether to enable it or not.

[Olly Betts]

I've now split off a xapian-letor top-level module.

[Olly Betts]

Merged the xapian-letor module to trunk in r16580, and working on avoiding needing to patch omindex.

Claiming this ticket, as I'm actively working on it.

[Olly Betts]

I've added detection of libsvm to configure, and also cleaned up a lot of the deviations from the coding conventions now.

[Olly Betts]

I'm not at all clear what the current state of things is with the letor branch, but it's time we got 1.3.1 out.

[Olly Betts]

We're not in GSoC 2015, so this isn't a blocker for more letor projects this year.

[Olly Betts]

It's time to get brutal for 1.4.0 - as a new module this could be introduced in a stable point release without problems.

[Olly Betts]

Thanks to this year's GSoC project the letor code is now on git master so "Merge learning to rank branch" is done, so we can close this ticket.

I think at this stage we don't try to backport this to 1.4.x, but instead focus on getting it into great shape for the next stable release series, which we're aiming to release much sooner after 1.4.0 than 1.4.0 was after 1.2.0.