In case of disk full, corrupt database

[Michiel Roding]

When running the scriptindex, and the disk runs full somewhere along the way - no errors are thrown and one ends up with a corrupt database (database '...' couldn't be opened)

[Olly Betts]

1.0.2 is pretty old - can you repeat this with 1.0.5?

Assuming yes:

You say "no errors are thrown", but what does happens? Does scriptindex exit, or hang or what?

And how large is the database? It would probably be useful to be able to inspect it.

[Michiel Roding]

The database is about 350mb; we run scriptindex by use of a popen() in PHP. I have not seen any errors in our log, though it may have not been picked up.

I have copies of the corrupt database, but as it contains sensitive information I cannot give it out. If I can in anyway run tests on it for you, please do ask.

[Richard Boulton]

Is this reproducible with 1.0.5? Or indeed, is it reliably reproducible with 1.0.2? Setting up a disk partition close to capacity is a little fiddly, so if you are in position to check for reproducibility with 1.0.5, that will save us a good deal of time.

A directory listing of the database directory might be interesting (with "ls -l", so we can see the modification times of files). Also, the output of running "xapian-check" on the database might be worthwhile.

Without being able to reproduce the problem, or inspect the database, it's unlikely we'll be able to help.

[Olly Betts]

Better still (assuming GNU ls):

ls -lrt --time-style=+%s.%N

If you don't have GNU ls, this might at least order files last modified in the same second:

ls -lrt

[Olly Betts]

Hmm, I wonder if this is due to the non-atomic updating of base files (already fixed in SVN)? If writing the base file fails due to disk-full, then maybe we end up with a partial base file.

If this is the problem, this patch should fix it:

​http://svn.xapian.org/branches/1.0/xapian-core/backends/flint/flint_table.cc?rev=10155&r1=10147&r2=10155&makepatch=1&diff_format=h

And the most recent base file will be smaller than it should be, probably noticeably so. Deleting this partial file will probably allow Xapian to open the database successfully - the database is probably "stuck" rather than really corrupted.

[Olly Betts]

Further information requested over a month ago, but no response from the reporter, so resolving this as "incomplete". If you have more information, please reopen.