Allow lazy document deletion

[Richard Boulton]

Currently, document deletion works by reading the termlist for the document, and updating the posting lists for each term immediately. Instead, we could keep a list of deleted document ids, and merge this list with the posting lists when iterating through them. This would allow deletion to be performed even when the termlist table is not present. The downside is that the statistics for the frequency of terms in the deleted documents would be incorrect.

Xapian-compact should be updated to merge the list with the database (ie, to apply the deletions) - this could probably be done with little extra overhead, since xapian-compact already needs to read through the posting lists.

[Olly Betts]

I think it's probably optimistic to think the overhead to fix this up in xapian-compact would be "little" - currently it copies chunks unmodified (except for a tweak to the start of chunks in some cases when renumbering documents). Having to unpack and repack them without some entries is going to add significant CPU use, though shouldn't change I/O much.

[Olly Betts]

Marking for brass rather than chert.

[Olly Betts]

Let's try to do this for honey.