Segmentation fault

[zangetsu]

I use xapian indirectly through "notmuch" and recently I observed segmentation faults for certain search queries using xapian 1.4.13, macOS 10.14.6.

After I edited the brew formula to reinstall 1.4.12 it didn't happen anymore.

I'm not sure how to provide a lot of information. I've added a little function to print the stack trace in notmuch, maybe it's a start:

% ./notmuch search from:foobar@
Error: signal 11:
0   notmuch                             0x000000010025d593 handler + 35
1   libsystem_platform.dylib            0x00007fff760d0b5d _sigtramp + 29
2   ???                                 0x00000000000007e0 0x0 + 2016
3   libxapian.30.dylib                  0x00000001008851d3 _ZNK14SelectPostList10get_weightEv + 33
4   libxapian.30.dylib                  0x000000010088b42f _ZN10MultiMatch8get_msetEjjjRN6Xapian4MSetERNS0_6Weight8InternalEPKNS0_12MatchDeciderEPKNS0_8KeyMakerE + 2393
5   libxapian.30.dylib                  0x00000001007c4083 _ZNK6Xapian7Enquire8Internal8get_msetEjjjPKNS_4RSetEPKNS_12MatchDeciderE + 615
6   libxapian.30.dylib                  0x00000001007c53ab _ZNK6Xapian7Enquire8get_msetEjjjPKNS_4RSetEPKNS_12MatchDeciderE + 25
7   notmuch                             0x000000010028f771 _notmuch_query_search_documents + 2673
8   notmuch                             0x000000010028ecf4 notmuch_query_search_messages + 36
9   notmuch                             0x000000010029034a notmuch_query_search_threads + 138

Else let me know what I can do to get more useful debug data.

I also uninstalled it via brew and cloned the github mirror, then ran git bisect: cba0c1f11338c99721882089ab98541f0e375b43 was the first commit to introduce this segfault between tags v1.4.12 and v.1.4.13.

[Olly Betts]

You've narrowed down what broke this, which is very helpful.

But I don't currently see what's broken here - this optimisation effectively should just be like rearranging the query.

Can you run this to get the query which notmuch is actually running:

NOTMUCH_DEBUG_QUERY=1 ./notmuch search from:foobar@

Also, if it's easy to do it would be useful to get a backtrace with debug info for libxapian available to confirm the line numbers where this is happening.

[Olly Betts]

Oh, I think I'm starting to see - this optimisation can make the query directly under a phrase check one which can prune, which couldn't happen before and the phrase check expect that. I don't see the full path to a NULL dereference from there, but it's probably via the object pointed to having been deleted.

The NOTMUCH_DEBUG_QUERY output would probably be useful still.

[Olly Betts]

I managed to reproduce this and fixed my reproduced version in f4ed23d13267c5cacf0626fbab89209b2e891848.

I'm confident I found the same problem you hit, but if you could confirm this also fixes your segfaults that would still be useful.

[zangetsu]

Ok, I got two queries where it segfaulted. I've adjusted the tag names and values, but the rest is faithful. Also note that the "foobar" of the first example is not the same as the one in the second.

NOTMUCH_DEBUG_QUERY=1 ./notmuch tag +foobar to:foo-bar@
Query string is:
( to:foo-bar@ ) and (not tag:foobar)
Exclude query is:
Query()
Final query is:
Query((Tmail AND ((XTOfoo@1 PHRASE 2 XTObar@2) AND (<alldocuments> AND_NOT Kfoobar))))

NOTMUCH_DEBUG_QUERY=1 ./notmuch search custom:one.two.three.four AND NOT tag:foobar
Query string is:
custom:one.two.three.four AND NOT tag:foobar
Exclude query is:
Query((Kdeleted OR Kspam))
Final query is:
Query(((Tmail AND ((XUcustom:one@1 PHRASE 4 XUcustom:two@2 PHRASE 4 XUcustom:three@3 PHRASE 4 XUcustom:four@4) AND_NOT Kfoobar)) AND_NOT (Kdeleted OR Kspam)))

These were two specific things that kept failing, however. All of my other tagging rules (the vast majority) still worked fine, so it's possible that some very specific things came together in some emails.

I built xapian with -g, but the resulting stack trace is just roughly the one from my initial report. I can't easily use gdb here, due to SIP, but let me know if there's an easy way to generate that backtrace that I'm missing.

[Olly Betts]

OK, both those examples fit the pattern of bug I fixed. Since 1.4.13, those get internally handled as:

intersect (AND-ed terms) -> exclude (OR of NOT-ed terms) -> phrase check

But that means the phrase check now needs to handle pruning (which happens if the NOT-ed terms run out before the AND-ed terms, to short-cut the now redundant exclude check).

(And for posterity, the code on master doesn't have this bug.)