Chert writable databases should take a parameter indicating flush threshold.

[richard]

Possibly, this should be a global parameter (ie, applies to all databases), or maybe it should be a database specific parameter (ie, set as a parameter to the "open" method for chert writable databases).

In any case, the current way of setting a flush threshold (ie, setting an environment variable) is unsatisfactory, due to being difficult to set in some circumstances (or on some OSes), and it being easy for users to forget to export the variable, resulting in bogus bug reports. A parameter to a Xapian function would be a cleaner API for this. However, we intend to improve the handling of automatic flushes in future, such that the count of added documents won't be the crucial factor; instead, amount of memory used will be. We need to ensure we don't add a parameter to the API which will shortly become meaningless.

[olly]

More thoughts:

Perhaps the parameter should be modifiable for an existing WritableDatabase? object - sometimes usage patterns change for an existing object and it would be annoying to have to close and reopen just to change the parameter.

And ideally we probably want a "how aggressive to be" measure - e.g. from 0 for "minimise resource usage" to 1 (or 10 or 100 or 1000 or whatever) for "this machine only exists to service your every whim".

[olly]

There's one issue with this that has just struck me - it's likely to make some bugs harder to reproduce if when we flush depends on how much memory is in use by other processes at the time. I'm not saying we should reject the idea, but perhaps we should try to design in some behavioural stability - e.g. if we check free memory we could round the result to the nearest N megabytes.

[richard]

That's a very good point; rounding will help, though maybe not very much. I suppose we could also introduce an environment variable to specify how much memory to use, to get repeatable test runs.

[richard]

Changed this bug to be about Chert - we want to put new features like this into Chert rather than Flint now.