Make autoflush in implicit transaction not commit.

[Richard Boulton]

Currently, when no explicit transaction is in progress, and an automatic flush occurs, the changes so far are committed.

Instead, it would be better if a flush of the buffered changes occurred, but not commit() happened.

This would lead to more predictable behaviour from the API (in particular, if a lot of documents are added without a flush, currently, a user may see that some of them are available for searching, and not realise that an explicit flush() is needed to make them all available).

This would probably be easiest to implement in conjunction with the changes to support #266, so I've marked this as blocked by #266.

[Olly Betts]

On the flip side, people might see their database getting bigger and bigger, and have heard that Xapian allows searching while indexing, and wonder why they can't see any (new) documents. Especially as this is the current behaviour.

True, documentation could explain this, but then documentation currently says that changes are flushed automatically every 10000 documents (which your hypothetical user apparently hasn't read).

Also, committing changes in larger batches will tend to bloat tables (especially the postlist table, but also others if documents are being updated) with unused blocks - the number of such blocks grows as the batch size does.

Your suggested behaviour can currently be achieved by using a transaction. With this change, getting the current behaviour isn't so easy. At present, you could count and commit explicitly every N changes, but that wouldn't allow the user to emulate more sophisticated flush criteria.

[Olly Betts]

I'm still not sold on this change, though if we're making it, 1.1.0 would be better than mid 1.1.x (but mid 1.1.x wouldn't actually be a big problem).

[Olly Betts]

I'm going to push this back to milestone:2.0.0 as this is a change which is likely to catch out existing users and there are pros and cons to both the current and proposed behaviour (so this isn't a bug fix or a clear enhancement). I'm leaning towards "won't fix" but it might make more sense in the scope of a more sweeping rethink about how transactions and committing changes are presented.

[Richard Boulton]

For the record, I'm leaning towards "won't fix" for this ticket too, based on Olly's analysis in comment #1 above.

[Olly Betts]

OK, let's go for that then. Seems I needs to unset blocked-by first.