omindex should make use of O_NOATIME where available

[Olly Betts]

On Linux >= 2.6.8, open() accepts a O_NOATIME flag which is intended for use by "indexing or backup programs". That means us!

I have a patch for this, which I'll attach shortly.

There's a wrinkle though - in some cases O_NOATIME will cause open to fail with EPERM and you need to retry the open call without O_NOATIME:

       EPERM  The  O_NOATIME  flag was specified, but the effective user ID of
              the caller did not match the owner of the file  and  the  caller
              was not privileged (CAP_FOWNER).

So for example, if we're indexing /usr/share/doc as a non-root user, we incur an extra syscall for each file - in this case it would be more efficient not to use O_NOATIME at all.

We need to quantify this overhead, and (if it's an issue) look at how to reduce it. One thought I had was, on a per-directory basis, to give up on using O_NOATIME if we failed to open a file using it. Then we only incur one syscall per directory for a read-only tree. Various tweaks to this are possible - e.g. give up for this directory and all subdirectories.

[Olly Betts]

Patch which adds O_NOATIME support

[Olly Betts]

For Unix, we can look at st_dev in struct stat to determine if this is the same FS - O_NOATIME is likely to apply at the FS level.

[Olly Betts]

We should at least consider this for 1.1.0.

[Olly Betts]

The patch isn't ready to apply and this isn't an incompatible change so postponing to 1.1.1.

[Olly Betts]

Triaging milestone:1.1.1 bugs.

[Olly Betts]

Not obvious how best to resolve this, and it's not an incompatible change, so bumping.

[Olly Betts]

Ubuntu now defaults to mounting partitions relatime, which means the atime doesn't get updated in most cases. I've not looked at what other distros do, but I suspect it's increasingly common as atime updates tend to kill I/O performance.

We could just check the euid of the process, and see if it is 0 or equal to the owner of the file (in omindex we've already called stat() on the file at the point where we read it, so we can check the owner for free) - if either is true, then we try O_NOATIME.

That seems simpler and more robust than caching failure per directory or FS. It misses the case where the process has CAP_FOWNER but isn't root, and doesn't own the files being indexed, but we just fail to try O_NOATIME in this case which means it would be handled just as it is now.

[Olly Betts]

Updated patch which checks the euid and file owner

[Olly Betts]

Applied a slightly updated version of the latest patch as r15143.

Timed tests didn't show a speed difference (or if there is one, it's smaller than the variations in indexing time) but it may be faster in some circumstances, and should never be slower, and it's also a bit unhelpful for indexing to change the atime.