Keep track of last modification time of database

[Richard Boulton]

It would be useful to be able to find out the time at which a modification was last made to a database. Two possible uses for this information:

• supporting caching in an HTTP search server: for example, this would allow if-modified-since requests to be satisfied without having to perform a search (when not modified since).

• displaying status information about a database to users.

To implement this, we'd need to store the modification time in the database somewhere, since we want to know the time of the last successful commit; this can't be derived by looking at the timestamp of the files, since it's possible that a commit nearly happened, but was cancelled or failed at the last minute (or, indeed, since we use fdatasync(), that a commit succeeded, but the timestamp information about the file modification never got written to disk due to a machine crash immediately after the commit). [This is true for chert and older backends, but for glass a new iamglass file is written to a temporary name and then renamed to atomically perform the commit, so its timestamp should be reliable even in the face of these concerns]

This could be emulated by storing the current time in a metadata value each time that flush() is called, but this wouldn't cover modifications made by implicit flushes. Currently, to implement this correctly on top of xapian, you'd need to use explicit transactions.

[Olly Betts]

Not sure I buy the status information use really - I kind of feel you're scratching around for a second use there!

But a way to determine if a previous query might return different results now would be handy. Not just for the HTTP if-modified-since, but for general caching of results - e.g. you could cache rendered output from Omega along with the timestamp. Hmm, actually just being able to read the current revision would be enough for that, but that doesn't seem to provide an easy way to implement the HTTP if-modified-since case. But using a timestamp is a bit brittle if the HTTP server and Xapian server are on different machines as there's potential for clock skew.

Acutally, isn't the timestamp of the newer base file of the record table a reliable timestamp, even in the face of failed updates? It won't be quite right after xapian-compact as record isn't updated last there, but it will just be a bit too old. The major advantage of this approach is that it will work for existing databases, both flint and chert.

[Richard Boulton]

Actually, the status information request came from Tom, who wanted to be able to display it for a set of databases which he'd been working on - if I understood him correctly, there were several of them, with similar data but indexed in different ways, and displaying the last modification time would have been a useful check on which one was which. There are probably other, better, ways of making that clear (database uuids, for starters), but I didn't make the request up. ;-)

The timestamp of the newer base file might work normally - though it would change when databases were replicated.

Having an accurate timestamp would also be useful for HEAD requests - a search server could handle a HEAD request for a search without doing the search (and include a last modified time header in the response).

[Olly Betts]

I think we want to avoid putting timestamp information into the database itself.

For the actual case here (checking manually which database was created when) ls -lt /PATH/TO/DB|head works well.

For HTTP "If-Modified-Since" and similar cases, I think the best option is a method which (in a backend-specific way) gets a suitable timestamp from the database files, or else returns (time_t)-1. Copying or replication may touch that timestamp, but I don't think that's much of an issue for the uses we're anticipating (and replication could easily replicate the file timestamps too).

The big plus of this approach is it works for all existing disk-based databases, and could be implemented if someone wrote a backend for a "foreign" format (say, a Lucene backend).

[Dan]

I'm going to chime and point out that even if we are replicating, the changesets will be also be created with monotonically increasing timestamps. So even if they are not the same as the main server, they should be in order and thus be of use for testing if a database has been modified since a given date. I think if we do start to store timestamps we'll have to deal with locality issues to get it right.

[Olly Betts]

Reviewing this, I'm still leaning towards file timestamps as the best approach overall. If the limitations prove to be a problem, we could always switch to storing an explicit timestamp later.

I think this probably should work on a Database object (rather than just being some static method taking a path).

I'm not entirely sure what the returned type should be. The obvious choices are time_t (1 second resolution), struct timeval (1 µs resolution), struct timespec (1 ns resolution) or double (which should be able to hold any 64-bit integer value exactly, but seems a little problematic). POSIX and C11 specify struct timespec, which makes it a promising option. It would mean pulling in an external system header though.

Getting the timestamp:

• For chert, we can look at the base file we used to open the record table.
• For glass, we can just use the timestamp of the version file.
• For inmemory, we'd probably just return unknown.
• For remote, this would need a protocol tweak (but could return unknown for now).
• For multidatabases, we'd return max() for the subdatabases (or unknown if any are unknown).
• For writable databases, I'm not really sure what's best. The options I can see are:
  • Define that's this is the last time a commit happened.
  • Update a timestamp on each modification (which is an extra overhead to maintain data we will probably never use for a case where the timestamp probably isn't very important anyway).
  • Return unknown.
  • Return the current time.

I don't see this as a blocker for 1.4.0, as it doesn't need incompatible changes, and nobody's asked about it since the initial request.

[Olly Betts]

Milestone renamed

[James Aylett]

I think it's worth rethinking whether we actually need this now we have database UUIDs and we expose the revision.

• I've just been debugging a problem which came down to not committing aggressively enough, and I didn't need more than revision to do that.
• UUIDs + revision can be used (with relevant HTTP parameters) to construct an ETag and using If-None-Match instead of doing If-Modified-Since.
• If you *really* need a modification time for user display purposes (eg "searching data up to DDDD/TTTT") then the database revision data isn't necessarily what you actually want, because your indexer may run at time T but miss documents in a window before it, or indeed may be indexing an archival corpus from 40 years ago or whatever. Such a display I think is always going to be about the semantics rather than the mechanics of the database (even though sometimes a database modified date is a reasonable proxy), so I think that better belongs in metadata managed by the indexer.
• For Tom's original need, which was niche, you can "just" take the modified timestamp on the base file (and do this outside Xapian).

[Olly Betts]

Ah yes, the database revision number provides an integer value which monotonically increases with each successful commit and avoids the various problems that a timestamp brings, so seems a more satisfactory basis for caching.

So really we're down to Tom's reporting tool use case - it would be slightly easier to write such a tool if there was an API method but it's not too hard to just look at the timestamp of iamglass (or other files for other backends) and nobody else has expressed interest in 18 years so it seems reasonable to mark this as "wontfix".

Maybe we should add a note on using the UUID and revision for caching somewhere though? Or perhaps Omega should provide some support for doing so.

[Olly Betts]

Reassigning as Vaibhav is clearly not actively working on this.

[Olly Betts]

Replying to James Aylett:

• UUIDs + revision can be used (with relevant HTTP parameters) to construct an ETag and using If-None-Match instead of doing If-Modified-Since.

I had a quick look at ETag - it seems Omega could easily emit a suitable HTTP header encoding the UUID and revision number for a single local database, e.g.

ETag: W/"b898b987-1fb4-4551-9966-e286e2cbd7e6@1234"

It's a bit trickier if we're searching multiple shards as get_revision() only works for a single shard, but I think we can iterate over the shards and get the revision of each. Remote databases would need changes to xapian-core.

We could add a method to the C++ API to generate a suitable string, which omega could then use.

I think it needs to be weak (W/) as we can't trivially guarantee the result is byte-for-byte identical. E.g. $now, $time, $random would result in non-identical output, the template could have changed, or a newer Xapian version might return different estimates of the number of matches. To generate a non-weak ETag we'd need to encode a hash of the template contents, Xapian version and check for use of non-reproducible OmegaScript commands.

I'm not sure how If-Match: would be usefully checked though. We can't leave it to the web server as it would need to run the CGI to get the ETag: header to compare, so we still need to do most of the work (we'd just avoid sending the result over the network if it hasn't changed). Omega could check this header if the webserver passes it through unchecked.

(Perhaps Omega should use something more modern than CGI...)

[James Aylett]

I suspect we don't need a non-weak ETag; it's intended for range requests (which in theory might be useful, but in practice probably wouldn't for most Xapian applications) and preventing lost updates on PUT (using If-Match, so you only replace the thing you expect to replace).

Supporting If-None-Match would be for conditional cache updating, which is probably the most useful. I believe that a weak etag there would be fine. (If we did want a strong etag, we could hash the generated body of the entity, which costs a buffer the size of the entity, which in most Xapian applications probably isn't too painful. But shouldn't be needed.)

We could add a weak ETag to omega to show the way. (The problem with replacing it with a non-CGI approach is that anything we choose is likely to become a dependency that shifts over time, increasing our maintenance burden.) It feels like a good place to start would be a Database call that generates something whether it's a direct db or a multidb, although probably just documenting that somewhere would be enough.