Chert should write all base files, then flush all base files

[Olly Betts]

OS X supports the F_FULLSYNC ioctl, other systems don't (currently, at least AFAIK). Xapian uses F_FULLSYNC if it is available to ensure that changes have actually made it to disk:

​http://trac.xapian.org/ticket/288

On other systems, it uses fdatasync() or fsync() which ensure that the data has at least left the OS, but it may only be in the controller or drive cache for potentially several seconds more.

Currently Xapian writes the base file for the table, then syncs it, then does the next table. I bet it would be more efficient to write them all and then sync them all, especially with F_FULLSYNC (hence I'm marking this as "Mac OS X").

I'll take a look at doing that. It shouldn't be hard.

[Richard Boulton]

Currently, Xapian iterates through the tables, calling a sync method first on the .DB file, then on the base file.

We should probably be syncing at two points per transaction: once after writing all the changes to the DB files and writing all the new base files except for the record table, and once after writing the record table base file.

[Richard Boulton]

I don't think this is a critical problem, or affects the API / ABI, so could probably be punted to 1.2.x.

[Olly Betts]

Indeed.

[acid2]

Hi, just like to add a little bit of information about my system here. I use notmuchmail.org with svn HEAD (at time of writing) of xapian-core. By disabling F_FULLSYNC with this regex:

perl -pi -e 's/^#ifdef F_FULLFSYNC/#if 0/' backends/*/*_io.h

The time to complete a basic operation drops by an order of magnitude from 0.7s to 0.06s. If there is any actual useful information I can give you, please let me know - I'm happy to provide anything I can, or be a test case for patches.

[Olly Betts]

Note that you shouldn't extrapolate from the overhead on a "basic operation".

The drive's writeback buffer is a fixed size, so the time to write out all the data from it to the disk platters is bounded, so the delay incurred by each commit by using F_FULLFSYNC is likely to be at most a few seconds even if you're adding a million documents to the database in that commit.

So the overhead of the F_FULLFSYNC is going to be proportionally much worse for a "basic operation" (changing a single tag and committing it) than when changes are batched up. And batching up changes is recommended for other reasons if you want good indexing performance.

[Olly Betts]

Brass now only has a single base file for all tables, and syncs all the table and then the base file after writing them all (at least on the brass-btree branch, as of r14086), so marking this ticket for chert.

[Olly Betts]

A few relevant updates:

As of 1.2.19 (and 1.3.2), we only full sync once per commit, which should still give the same guarantees as full syncing all the files (unless you move the tables elsewhere and symlink them in, or similar odd cases).

1.3.2 defaults to not using F_FULLSYNC, and provides a Xapian::DB_FULL_SYNC flag which can be passed to get full-sync where available (which seems to be only OS X currently).

And this is only a performance issue, so I don't think it makes sense to change flint at this point - 1.2.x defaults to chert, and supports flint mainly to allow people to easily migrate.

I would be interested to know how 1.2.19 or later performs on OS X in practice.

[Olly Betts]

Close to 8 months on and no explicit feedback sadly.

But I've also not heard any complaints about commit performance, so I'm going to assume the changes in 1.2.19 helped, and I'm going to mark this as fixed (especially as it has definitely been addressed for the glass backend in 1.3.x). If anyone is still seeing issues, please comment with details and we can reopen.