Add support for SCWS Chinese segmentation library

[Olly Betts]

Discussion:

​http://thread.gmane.org/gmane.comp.search.xapian.general/9047

Latest patch:

​http://thread.gmane.org/gmane.comp.search.xapian.general/9052

[Olly Betts]

Updated patch from second email thread

[Olly Betts]

Updated patch which compiles without SCWS

[Olly Betts]

I've cleaned up the patch a little, and fixed some warnings so it compiles cleanly without SCWS. I've not tried it with SCWS yet.

[Olly Betts]

There's a newer patch, but not based on my cleaned up version:

​http://article.gmane.org/gmane.comp.search.xapian.general/9359

I'll also attach the patch file so it can't get lost.

[Olly Betts]

newer patch from original author

[Olly Betts]

Unfortunately, I think we have to bump this from this release cycle. It's really the sort of change that should go in early in the cycle so there's ample time to shake out problems, not right at the end which is where we now are.

I should have sorted this out earlier in this cycle, but frankly the updated patch which totally ignored the work I'd already done cleaning up the previous one was quite a demotivator for spending time on this.

Anyway, let's get this in early in 1.5.x (which we don't yet have a milestone for so 1.4.x for now).

[Olly Betts]

We now have support for CJK segmentation using ICU (merged in [f881f0bd1609]).

The patches here are all sadly very outdated (mostly my fault). But I think at this point closing this ticket makes the most sense.

If SCWS does a better job for Chinese than ICU we could potentially support both (and indeed other segmentation algorithms). I think for maintainability additional alternatives would each needs to be wrapped up cleanly in an iterator class in the same way that CJKNgramIterator and CJKWordIterator wrap the ngram and ICU algorithms.