Opened 17 years ago

Closed 18 months ago

#195 closed defect (duplicate)

Writable databases should take a parameter indicating flush threshold.

Reported by: Richard Boulton Owned by: Olly Betts
Priority: normal Milestone: 2.0.0
Component: Backend-Glass Version: git master
Severity: normal Keywords:
Cc: Blocked By:
Blocking: Operating System: All

Description (last modified by Richard Boulton)

Possibly, this should be a global parameter (ie, applies to all databases), or maybe it should be a database specific parameter (ie, set as a parameter to the "open" method for chert writable databases).

In any case, the current way of setting a flush threshold (ie, setting an environment variable) is unsatisfactory, due to being difficult to set in some circumstances (or on some OSes), and it being easy for users to forget to export the variable, resulting in bogus bug reports. A parameter to a Xapian function would be a cleaner API for this. However, we intend to improve the handling of automatic flushes in future, such that the count of added documents won't be the crucial factor; instead, amount of memory used will be. We need to ensure we don't add a parameter to the API which will shortly become meaningless.

Change History (14)

comment:1 by Olly Betts, 17 years ago

More thoughts:

Perhaps the parameter should be modifiable for an existing WritableDatabase object - sometimes usage patterns change for an existing object and it would be annoying to have to close and reopen just to change the parameter.

And ideally we probably want a "how aggressive to be" measure - e.g. from 0 for "minimise resource usage" to 1 (or 10 or 100 or 1000 or whatever) for "this machine only exists to service your every whim".

comment:2 by Olly Betts, 17 years ago

Owner: changed from New Bugs to Olly Betts

There's one issue with this that has just struck me - it's likely to make some bugs harder to reproduce if when we flush depends on how much memory is in use by other processes at the time. I'm not saying we should reject the idea, but perhaps we should try to design in some behavioural stability - e.g. if we check free memory we could round the result to the nearest N megabytes.

comment:3 by Richard Boulton, 17 years ago

That's a very good point; rounding will help, though maybe not very much. I suppose we could also introduce an environment variable to specify how much memory to use, to get repeatable test runs.

comment:4 by Olly Betts, 17 years ago

Operating System: All
Status: newassigned

comment:6 by Richard Boulton, 16 years ago

Component: Backend-FlintBackend-Chert
Description: modified (diff)
Summary: Flint writable databases should take a parameter indicating flush threshold.Chert writable databases should take a parameter indicating flush threshold.

Changed this bug to be about Chert - we want to put new features like this into Chert rather than Flint now.

comment:7 by Olly Betts, 12 years ago

Component: Backend-ChertBackend-Brass
Summary: Chert writable databases should take a parameter indicating flush threshold.Writable databases should take a parameter indicating flush threshold.

Updating to be about brass, and marking to consider for 1.3.x.

comment:8 by Olly Betts, 12 years ago

Milestone: 1.3.x

Oops, I failed to actually set the milestone with my previous comment.

comment:9 by Olly Betts, 11 years ago

Milestone: 1.3.x1.3.3

comment:10 by Olly Betts, 11 years ago

See also #266.

comment:11 by Olly Betts, 10 years ago

Component: Backend-BrassBackend-Glass

comment:12 by Olly Betts, 10 years ago

Milestone: 1.3.31.3.4

comment:13 by Olly Betts, 9 years ago

Milestone: 1.3.41.4.x

I don't have good answers to the issues here, so postponing.

comment:14 by Olly Betts, 20 months ago

Milestone: 1.4.x2.0.0
Version: SVN trunkgit master

comment:15 by Olly Betts, 18 months ago

Resolution: duplicate
Status: assignedclosed

I just spotted we also have #266 which covers this and more, so closing this ticket as a duplicate.

Note: See TracTickets for help on using tickets.