Indexing characters outside the BMP can result in exception

[Richard Boulton]

A document containing certain unicode characters can result in a zero length termname being generated by omega's query parser, which causes Xapian to throw an "Empty termnames aren't allowed." error.

The minimal example data file I've found is simply a document containing only the character 0x28a0f, which is a CJK Unified Ideograph. I have a fix, which I will commit shortly, which simply checks that the termlength isn't zero before adding it to the document, but it's possible that we should be fixing this by generating a term containing the unicode character, instead of just throwing it away.

[Richard Boulton]

Minimal test data file

[Olly Betts]

Actually it's not CJK, but rather a "Unified Han Ideograph" according to the Unicode charts. Note that this is outside the BMP, so not supported by the unicode routines we're using (taken from Tcl).

However, we shouldn't ever generate a zero length term even when presented with characters outside the range we currently handle.

[Olly Betts]

I've been mulling this over - I think the best answer for now is probably to assume any characters outside of the BMP are word characters. Eventually we ought to sort out unicode routines which handle such characters fully, but I think it's more important to focus on getting 1.0 out with very good unicode support than try to aim for perfect unicode support and delay the release.

What was the document you encountered this character in? Did it just have a few characters outside the BMP?

[Richard Boulton]

Yes, IIRC the document I encountered this in had only this one character outside the BMP: I'm afraid I don't have the actual document to hand.

Treating it as a word character seems the right approach for now to me, too. In this particular example, it wouldn't have mattered particularly whether the character could be found - it just mattered that it threw an error.

[Olly Betts]

OK, I've now committed a fix which assumes characters outside the BMP are word characters, but can't be forced to lowercase.

And looking at the check for zero length terms, that shouldn't be required because we should never be able to arrive at the check with an empty term. But we do, even with the fix above.

The real cause of this bug is that my utf-8 code was decoding 4 byte sequences incorrectly, so when we try to convert a too-large value back to utf-8, we get an empty string. I've committed a fix for this incorrect decoding, so this is fixed in SVN HEAD.

[Olly Betts]

Fixed in 1.0.0 release, which now knows about characters outside the BMP.