Ctrl-ing/killing scriptindex will leave you with an un-updatable database...

[arjen]

When one has an indexing process (scriptindex in my case) running, which gets for some reason killed - either due to a system crash or manually - the _baseA-files are lost and scriptindex won't be able to re-update the database...

[olly]

_baseA doesn't have to exist, but at least one of _baseA or _baseB needs to (and killing the indexer shouldn't be able to leave both missing).

What's the error message you get?

[arjen]

I get the following exception: Exception: New revision too low

And my database looks like: drwxr-xr-x 2 root root 4.0K Sep 29 05:31 ./ drwxr-xr-x 4 acm users 83 Sep 29 07:21 ../ -rw------- 1 root root 0 Sep 29 05:31 db_lock -rw-r--r-- 1 root root 10 Jun 30 23:04 meta -rw-r--r-- 1 root root 6.4G Sep 27 16:53 position_DB -rw-r--r-- 1 root root 102K Sep 27 16:51 position_baseB -rw-r--r-- 1 root root 3.3G Sep 27 17:12 postlist_DB -rw-r--r-- 1 root root 53K Sep 27 16:51 postlist_baseB -rw-r--r-- 1 root root 318M Sep 27 16:53 record_DB -rw-r--r-- 1 root root 5.0K Sep 27 16:51 record_baseB -rw-r--r-- 1 root root 2.8G Sep 27 16:53 termlist_DB -rw-r--r-- 1 root root 45K Sep 27 16:51 termlist_baseB -rw-r--r-- 1 root root 69M Sep 27 16:53 value_DB -rw-r--r-- 1 root root 1.1K Sep 27 16:51 value_baseB

[olly]

Has this database been around a while?

I wonder if the revision numbers have wrapped around!

What do you get from: quartzcheck DATABASEDIRECTORY

[arjen]

No, it's not around *that* long I think. It's a fresh database from when 0.8.0 was released and we mostly update it with batches of 1000 documents. With almost 1M documents, that results in an revision count of somewhat above 1000. And it does display a revision number of 1017 when I run quartzcheck.

I'll leave quartzcheck running to see if it can find anything wrong in the database backup, but I would be surprised if it did, since a 0.8.1-version didn't report any problems before I stored it away.

[arjen]

Besides, that assumption seems to be incorrect with the noticed behaviour. Since it will continue its indexing process if you don't kill it (and that means doing a batch of 1000 documents, preparing a new batch starting a new scriptindex-invocation, etc). And the revision number should be the same when the process restarts after it got killed during indexing, shouldn't it?

[olly]

Are all 5 tables at revision 1017?

[olly]

I managed to reproduce this here while looking at another problem!

There was a problem calculating the new revision number in this situation. It should be fixed in CVS - this is the patch:

http://cvs.xapian.org/xapian/xapian-core/backends/quartz/btree.cc.diff?r1=1.156&r2=1.157

There's no database corruption involved, so no need to rebuild the databases you were having this problem with.

[olly]

Fixed in 0.8.4