QueryParser: Confusion with Boolean operator and -.

[Carl Worth]

Here's a query string handled just fine by the query parser:

tag:foo and tag:deleted

which expands (in notmuch) to:

Xapian::Query((Tmail AND 0 * Kfoo AND 0 * Kdeleted))

Meanwhile, this very similar query string (adding only -):

tag:foo and -tag:deleted

instead expands to:

Xapian::Query((Tmail AND (Zand:(pos=1) FILTER (Kfoo OR Kdeleted))))

Note that things have gone quite insane here, (with the Boolean operator now being treated as a term). This seems to happen whenever a Boolean operator appears adjacent to a word starting with a non-alphabetic character, (but I haven't characterized the bug perfectly yet).

The same bug occurs with the operator in all caps (AND).

Here are the flags that notmuch is passing to the query parser:

(Xapian::QueryParser::FLAG_BOOLEAN |

Xapian::QueryParser::FLAG_PHRASE | Xapian::QueryParser::FLAG_LOVEHATE | Xapian::QueryParser::FLAG_BOOLEAN_ANY_CASE | Xapian::QueryParser::FLAG_WILDCARD | Xapian::QueryParser::FLAG_PURE_NOT)

I'd love to hear any ideas, and especially a fix of course. :-)

-Carl

[Olly Betts]

Fix for the syntax error

[Olly Betts]

a AND -foo:b

This is currently a syntax error, so the QueryParser switches to a fall back mode of parsing it without boolean operators, love, or hate enabled. That certainly isn't helpful in this case.

As a workaround, you can use this query instead, since AND is implicit for filters:

a -foo:b

I've attached a patch, but although it's simple, the implications of it aren't so clear. It seems to fix this case, and the changes it makes to the parsing of existing testcases look like they are just different representations with the same meaning, so it's probably OK, and I'd like to hear if it works for you.

Marking for 1.2.x for now, as this doesn't require ABI or API changes.

[Carl Worth]

Replying to olly:

As a workaround, you can use this query instead, since AND is implicit for filters:

a -foo:b

Thanks for the suggestion. As it is, I'm somewhat paranoid about always explicitly adding Boolean operators because I haven't yet found a way to get Xapian to always do the right thing with various prefixes, (see defect #245 for an example). Once that's fixed it will be quite nice to start typing queries without explicit operators.

I've attached a patch, but although it's simple, the implications of it aren't so clear. It seems to fix this case, and the changes it makes to the parsing of existing testcases look like they are just different representations with the same meaning, so it's probably OK, and I'd like to hear if it works for you.

I can't comment on any of the implications, but the patch does seem to solve my problem. So thanks for the quick fix! I'll look forward to this landing in Xapian.

-Carl

[Olly Betts]

OK, I've looked at the patch more closely, and it looks good with a tweak for an error message. My current version of that tweak is leaking an object each time, and I need to go out, but I'll sort that out and apply later or tomorrow.

Since there's a patch almost ready, marking this for 1.1.4.

[Olly Betts]

Fixed in trunk by r13990, and backported for 1.0 in r13991.