spurious DatabaseCorruptError with boolean term query and sparse docids

[Dan Callaghan]

The Sup email client (​https://supmua.dev/) uses Xapian to index emails. Every email is indexed as a document with a Kmail term. Sup also uses an unusual docid assignment scheme: docids are assigned based on the email's date in reverse order, starting from the year 2038 with the docid increasing as the date goes further back in time to 1970. Sup then performs boolean queries in ascending docid order, to find most recent emails first.

This docid assignment scheme means that the docid numbering space is non-contiguous and spread over a wide range. Sup users may have millions of messages covering decades.

When the user asks Sup to list all mail, Sup performs a query for the Kmail term. When retrieving more than a certain number of results (in my testing around 400) Xapian raises a spurious DatabaseCorruptError, even though the database is not actually corrupt according to xapian-check.

This has been reported in Sup as: ​https://github.com/sup-heliotrope/sup/issues/628

The spurious exception is reproducible with some straightforward Xapian operations, using the unusual reverse docid assignment behaviour mimicking Sup:

#include <xapian.h>
#include <cstdio>
#include <cstdlib>

int main() {
    system("rm -rf /tmp/xapian-bug-repro-db");
    Xapian::WritableDatabase db("/tmp/xapian-bug-repro-db",
                                Xapian::DB_CREATE_OR_OPEN);

    const Xapian::docid TOPID = 2000000;
    const unsigned N_DOCS = 2000;
    for (unsigned i = 0; i < N_DOCS; i++) {
        Xapian::Document doc;
        doc.add_term("Kmail", 0);
        db.replace_document(TOPID - i, doc);
    }
    db.commit();

    try {
        Xapian::Enquire enquire(db);
        enquire.set_weighting_scheme(Xapian::BoolWeight());
        enquire.set_query(Xapian::Query("Kmail"));
        Xapian::MSet mset = enquire.get_mset(0, N_DOCS);
        printf("OK size=%u\n", unsigned(mset.size()));
    } catch (const Xapian::Error& e) {
        printf("CRASH: %s: %s\n", e.get_type(), e.get_msg().c_str());
        return 1;
    }
    return 0;
}

This fails with Xapian 1.4.31:

$ ./xapian-repro
CRASH: DatabaseCorruptError: Unexpected end of posting list for 'Kmail'