DatabaseError when interleaving reads and writes

[richard]

Alessandro Pasotti reported a problem to xapian-discuss on 23 April 2008, where some PHP code, using the clustering branch of xapian, was performing an indexing run and then a "cluster" operation on the same WritableDatabase? object without an intermediate flush. The cluster operation returned DatabaseError?: inflate failed (invalid distance too far back). He later sent me a testcase by private email which I was able to use to track down the problem, and work out a smaller testcase.

I was quickly able to reproduce the problem on SVN HEAD, and have now worked out a minimal testcase which reproduces the same problem, in a different way.

I have narrowed down the problem to the "next_for_sequential" method in FlintTable. This is called (via next()) by FlintTable.read_tag(), when reading the subsequent components of a multipiece tag, and should never return false. (We should probably have a check in FlintTable.read_tag() that next() doesn't return false, which raises DatabaseCorruptError? if it does).

However, when the next piece of a tag is in a later block than the current block, and that new block has been newly added (since the last flush, or the database was opened), next_for_sequential incorrectly returns false. If I comment out the call to next_for_sequential in next(), so that next_default() is always used instead, the code works correctly.

I believe the problem is that next_for_sequential checks whether a block is in the database by comparing its number with base.get_last_block(). However, the last_block member in base is only updated when a flush is called, so doesn't reflect newly added blocks. Thush, next_for_sequential returns false incorrectly.

This bug affects both Flint and Chert databases.

[richard]

Testcase which reproduces the problem

[richard]

Fixed in SVN HEAD in revision [10401]. I'm applying a backport to the 1.0 branch, and will update the clustering branch with recent changes in HEAD shortly to check that the fix works for the original testcase, too.

[richard]

Applied to branch1.0 now.