All-stopword queries with two or more terms should ignore stopword list

[Richard Boulton]

Currently, if a single word query is parsed, and that word is a stopword, the stopwording is ignored. However, if a multiple word query is parsed, and all words are stopwords, the stopwording is applied (resulting in an empty query).

If all the words in the query are stopwords, I think it may make sense to ignore the stopwording. However, even if we decide to apply the stopwording in this case, we should be consistent in our behaviour.

Some examples, in python:

>>> import xapian
>>> s=xapian.SimpleStopper()
>>> s.add('foo')
>>> s.add('bar')
>>> qp=xapian.QueryParser()
>>> qp.set_stopper(s)
>>> str(qp.parse_query('foo'))
'Xapian::Query(foo:(pos=1))'
>>> str(qp.parse_query('foo foo'))
'Xapian::Query()'
>>> str(qp.parse_query('foo bar'))
'Xapian::Query()'

Either the first parse_query() call should return Xapian::Query(), or the later ones should return non-empty queries.

[Olly Betts]

The bug is that all-stopword queries with two or more terms aren't handled specially - the single term case was easy to handle, while the multi-term case seemed trickier for some reason, and I only handled the easy one so far.

There's actually a FIXME comment about this in queryparser.lemony which probably shows where the fix should go:

FIXME what if E && E->empty() (all terms are stopwords)?

[Olly Betts]

Simple approach to fixing this

[Olly Betts]

Marking for 1.2.x.

[Olly Betts]

The approach in the patch seems reasonable to me, so I'm going to take another look at this and if I don't see a better fix, I'll update and apply it for 1.2.3.

[Olly Betts]

There's a case the patch doesn't handle: the AND an

This doesn't work currently either, so the patch is an improvement, but it would be better to solve this whole issue in one go if we can.

[Olly Betts]

Actually the AND an works - the failing testcase is: the AND an a

Which is pretty unlikely to be hit in practice unless the stopword list is rather long, but the error message is confusing.

[Olly Betts]

More sophisticated fix, but a bit ugly

[Olly Betts]

New patch seems to fix all the cases I've checked without breaking more, but feels rather clumsy in places. It's bedtime now, but I'll have a look when I'm more awake and see if it can be done more elegantly.

[Olly Betts]

I committed the patch (with comment improvements) in trunk r14845. Maybe there's a neater way, but it does at least work.

I think this probably isn't worth backporting to 1.0 at this point - I've not seen any feedback from end users on this (unless that was what triggered your report?)