Query parser crash with NEAR and default op of AND_MAYBE

[Richard Boulton]

As reported at ​http://thread.gmane.org/gmane.comp.search.xapian.general/8562 the queryparser crashes when default op is set to OP_AND_MAYBE and the following query is parsed {{{ x-y NEAR test NEAR test }}}

According to the reporter, this is not a problem with 1.0.18::

With xapian-core 1.0.18, the same code yields an error: "Exception: Exception: Xapian::Query: AND_MAYBE requires a minimum of 2 and a maximum of 2 sub queries, had 4"

(I have not verified this.)

[Richard Boulton]

Patch with testcase reproducing this problem

[Richard Boulton]

The attached testcase causes queryparsertest to fail for me with a segfault when applied to trunk r15139.

valgrind indicates that the problem is a double free of a TermGroup object:

Running test: qp_and_maybe_near...==14053== Invalid read of size 4
==14053==    at 0x41B7CF9: yy_destructor(yyParser*, unsigned char, YYMINORTYPE*) (stl_iterator.h:686)
==14053==    by 0x41B7FCA: yy_pop_parser_stack(yyParser*) (queryparser_internal.cc:2049)
==14053==    by 0x41C19D6: ParserHandler::~ParserHandler() (queryparser_internal.cc:2066)
==14053==    by 0x41BDA88: Xapian::QueryParser::Internal::parse_query(std::string const&, unsigned int, std::string const&) (queryparser.lemony:1186)
==14053==    by 0x41B4BCF: Xapian::QueryParser::parse_query(std::string const&, unsigned int, std::string const&) (queryparser.cc:118)
==14053==    by 0x8065293: test_qp_and_maybe_near() (queryparsertest.cc:2302)
==14053==    by 0x807B342: test_driver::runtest(test_desc const*) (testsuite.cc:329)
==14053==    by 0x807CCBD: test_driver::do_run_tests(__gnu_cxx::__normal_iterator<std::string const*, std::vector<std::string, std::allocator<std::string> > >, __gnu_cxx::__normal_iterator<std::string const*, std::vector<std::string, std::allocator<std::string> > >) (testsuite.cc:602)
==14053==    by 0x807D3A6: test_driver::run_tests(__gnu_cxx::__normal_iterator<std::string const*, std::vector<std::string, std::allocator<std::string> > >, __gnu_cxx::__normal_iterator<std::string const*, std::vector<std::string, std::allocator<std::string> > >) (testsuite.cc:565)
==14053==    by 0x807D401: test_driver::run(test_desc const*) (testsuite.cc:776)
==14053==    by 0x804E1B6: main (queryparsertest.cc:2349)
==14053==  Address 0x45070e0 is 0 bytes inside a block of size 16 free'd
==14053==    at 0x4024851: operator delete(void*) (vg_replace_malloc.c:387)
==14053==    by 0x41B922D: TermGroup::as_group(State*) const (queryparser.lemony:1390)
==14053==    by 0x41BA1DF: Parse(yyParser*, int, Term*, State*) (queryparser.lemony:1888)
==14053==    by 0x41BD567: Xapian::QueryParser::Internal::parse_query(std::string const&, unsigned int, std::string const&) (queryparser.lemony:1182)
==14053==    by 0x41B4BCF: Xapian::QueryParser::parse_query(std::string const&, unsigned int, std::string const&) (queryparser.cc:118)
==14053==    by 0x8065293: test_qp_and_maybe_near() (queryparsertest.cc:2302)
==14053==    by 0x807B342: test_driver::runtest(test_desc const*) (testsuite.cc:329)
==14053==    by 0x807CCBD: test_driver::do_run_tests(__gnu_cxx::__normal_iterator<std::string const*, std::vector<std::string, std::allocator<std::string> > >, __gnu_cxx::__normal_iterator<std::string const*, std::vector<std::string, std::allocator<std::string> > >) (testsuite.cc:602)
==14053==    by 0x807D3A6: test_driver::run_tests(__gnu_cxx::__normal_iterator<std::string const*, std::vector<std::string, std::allocator<std::string> > >, __gnu_cxx::__normal_iterator<std::string const*, std::vector<std::string, std::allocator<std::string> > >) (testsuite.cc:565)
==14053==    by 0x807D401: test_driver::run(test_desc const*) (testsuite.cc:776)
==14053==    by 0x804E1B6: main (queryparsertest.cc:2349)

I've not yet determined the cause of the double free, and have run out of time for today. I'll try and follow up further tomorrow.

[Olly Betts]

The problem occurs if Query construction throws an exception after we've deleted the TermGroup object. Then the stack unwinding deletes the parser stack, which still has the TermGroup object on it.

Fixed for trunk in r15140. Needs backporting for 1.0.23 (though the original reporter says his testcase worked there, the same issue appears to exist there too - presumably it just isn't triggered by his testcase).

[Olly Betts]

Backported in r15200.