Matcher could optimise by hoisting near/phrase filter

[Olly Betts]

The near/phrase filter could usefully be hoisted up the tree in some cases (leaving the AND part where it is). Consider:

e-mail AND filter This is probably more efficient as a search for: `e AND mail AND filter' with results from that filtered for phrase matches on "e mail".

But that's not clear cut. It might be that every document matches that AND, but just one match the phrase. In that case, the current code will try the phrase check on all documents, find one match, skip to that posting in "filter", find it matches, and return one results.

If the phrase match is hoisted, then the 3-way AND needs to look at *all* of the postings for "filter", and the phrase filtering still does the same amount of work.

This extreme is probably rare, but it's not totally obvious that hoisting the phrase filter is a good idea generally. Or it might be there's a good heuristic for when to. If there's an AND with a rare term which we can hoist the filter above for example...

[Olly Betts]

It would be nice to implement this in the 1.0 series.

[Olly Betts]

I have a working patch for this which I'm currently running performance tests on. This should make it into 1.0.4, so marking this bug for 1.0.4.

[Olly Betts]

(Finally) fixed in SVN HEAD. The savings are impressive, and I think the worries about making things worse in some corner cases have turned out to be unfounded, or at least swamped by the improvements in real world cases.

[Olly Betts]

Fixed in 1.0.4