Redo machinery in InMemory? backend to allow multierrhandler1 to work. Probably leave until user database backends are possible, then do it by subclassing InMemory?...

Implement a Xapian::Fields class to serialise/unserialise name=value pairs to/from Document data field.

Use chains of free blocks rather than a bitmap - then we can store many old revisions more cheaply (just the space they actually need, not a whole bitmap for each one too). Then readers use fcntl locking on a single byte corresponding to the revision they're using (bytes off the end of the file can be locked, and shared locks on read-only files are ok). Then a writer would only delete old revisions for which it could obtain an exclusive lock (otherwise it would preserve them).

The Btree manager is generally written with multiple old revisions in mind, so this shouldn't be a huge project.

Provide explicit support for range searches, such as "RangePostList?" - combine a sequence of adjacent terms...

Tinderbox isn't cleanly configurable in the way we want, so I've had to hack it around a lot. Buildbot looks a much better bet, as it's designed to allow modification by subclassing.

Check for zero byte cleanness wherever strings are used. There are a number of c_str()s in the code, but I believe all in the core library are harmless at 2002-04-29. There may be other zero byte issues though. xapian-applications/dbtools also uses c_str() where it should probably use data() and length(). xapian-bindings hasn't been checked.

We are having an issue on a testing machine. We are running stress tests on it, and Xapian eventually raises a RuntimeError?, "unknown Xapian error".

This is on Windows, using the Xapian Python bindings.

Richard mentioned that exception.i has a catch(...) that catches all the unknown exceptions.

He also mentioned that this might have something to do with Windows Structured Exceptions, and that Mark had investigated this previously so he thought it had been fixed.

The following returns an error, even though the right arguments have been passed:

$vrp = new XapianNumberValueRangeProcessor?(0, "\$", true); $vrp->apply((string)"240", (string)"500");

The error returned is: Fatal error: Type error in argument 2 of NumberValueRangeProcessor?_apply. Expected SWIGTYPE_p_stdstring in xapian.php on line 1217

line 1217 being: return NumberValueRangeProcessor?_apply($this->_cPtr,$begin,$end);

Have tried changing that line to say (string)$begin and (string)$end, same result.

I've observed this from python, but I expect it can occur from C++ too. The following python script gives an assertion error:

import xapian a=xapian.Query('A') b=xapian.Query('B') c=xapian.Query(xapian.Query.OP_OR, a, b) d=xapian.Query(xapian.Query.OP_NEAR, c, c)

The error is: xapian.AssertionError?: /home/richard/private/Working/xapian/xapian-core/api/omqueryinternal.cc:770: op

Xapian::Query::OP_NEAR || op

Xapian::Query::OP_PHRASE

This is because an attempt is made to flatten the query "(A OR B) NEAR (A OR B)", which isn't supported (I believe). It would be nice to fix this by supporting such searches, but meanwhile we shouldn't raise AssertionError? from the API; a more explanatory exception should be returned instead.

Building ubuntu dapper or gutsy gives a similar failure for command fakeroot apt-get source -b xapian-core.

g++ -DHAVE_CONFIG_H -I. -I.. -I../common -I../include -I./include

-I../languages -Ilanguages -I../queryparser -Wall -W -Wredundant-decls -Wpointer-arith -Wcast-qual -Wcast-align -Wno-multichar -Wno-long-long -fno-gnu-keywords -Wundef -Wshadow -fvisibility=hidden -O2 -c ../api/editdistance.cc -o api/editdistance.o >/dev/null 2>&1 /bin/sh ./libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -I../common -I../include -I./include -I../languages -Ilanguages -I../queryparser

-Wall -W -Wredundant-decls -Wpointer-arith -Wcast-qual -Wcast-align

-Wno-multichar -Wno-long-long -fno-gnu-keywords -Wundef -Wshadow -fvisibility=hidden -O2 -c -o api/error.lo ../api/error.cc

g++ -DHAVE_CONFIG_H -I. -I.. -I../common -I../include -I./include

-I../languages -Ilanguages -I../queryparser -Wall -W -Wredundant-decls -Wpointer-arith -Wcast-qual -Wcast-align -Wno-multichar -Wno-long-long -fno-gnu-keywords -Wundef -Wshadow -fvisibility=hidden -O2 -c ../api/error.cc -fPIC -DPIC -o api/.libs/error.o In file included from ../api/error.cc:25: ../common/safeerrno.h:25:3: error: #error You must #include <config.h> before #include "safeerrno.h" make[3]: *** [api/error.lo] Error 1 make[3]: Leaving directory `/home/rhatch/xapian-core-1.0.5/xapian-core-1.0.5/build' make[2]: *** [all-recursive] Error 1 make[2]: Leaving directory `/home/rhatch/xapian-core-1.0.5/xapian-core-1.0.5/build' make[1]: *** [all] Error 2 make[1]: Leaving directory `/home/rhatch/xapian-core-1.0.5/xapian-core-1.0.5/build' make: *** [build-stamp] Error 2 Build command 'cd xapian-core-1.0.5 && dpkg-buildpackage -b -uc' failed. E: Child process failed

Prior to this failure the command below does not seem to work. sudo apt-get build-dep xapian-core Reading package lists... Done Building dependency tree... Done 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Currently, if a single word query is parsed, and that word is a stopword, the stopwording is ignored. However, if a multiple word query is parsed, and all words are stopwords, the stopwording is applied (resulting in an empty query).

If all the words in the query are stopwords, I think it may make sense to ignore the stopwording. However, even if we decide to apply the stopwording in this case, we should be consistent in our behaviour.

Some examples, in python:

import xapian s=xapian.SimpleStopper?() s.add('foo') s.add('bar') qp=xapian.QueryParser?() qp.set_stopper(s) str(qp.parse_query('foo'))

'Xapian::Query(foo:(pos=1))'

str(qp.parse_query('foo foo'))

'Xapian::Query()'

str(qp.parse_query('foo bar'))

'Xapian::Query()'

Either the first parse_query() call should return Xapian::Query(), or the later ones should return non-empty queries.

I use xapian-core-1.0.7 with the corresponding perl bindings. I run a 1 writer/N reader setup, and I do reopen() a database-handle before each query. Nevertheless I casually get DatabaseModifiedErrors.

This is what I found out so far:

* The errors occurs after explicit flushing my most frequented index. The error does less often occur, if I do a sleep(1) after each explicit flush() before applying no changes (without flush) to the index, and it never occured so far with a sleep(4). This is my workaround.

* I already set XAPIAN_FLUSH_THRESHOLD to a large value (100000).

* I patched the xapian-core lib to log all calls of FlintDatabase::set_revision_number(), and the throw-points of the XapianModifiedErrors, which turned out that the exception gets thrown in FlintTable::set_overwritten().

* I patched again to get the caller and found out that set_overwritten() got called by FlintTable::block_to_cursor(), which I patched again to expose the condions:

if (REVISION(p) > REVISION(C_[j + 1].p)) {
 fprintf(stderr, "set_overwritten: from block_to_cursor() %d > %d\n", REVISION(p), REVISION(C_[j + 1].p));
 set_overwritten();
 return;
}

and it turned out:

set_overwritten: from block_to_cursor() 10194 > 10192
terminate called after throwing an instance of 'Xapian::!DatabaseModifiedError'
(...)
set_overwritten: from block_to_cursor() 10195 > 10193
terminate called after throwing an instance of 'Xapian::!DatabaseModifiedError' set_overwritten: from block_to_cursor() 10195 > 10193
terminate called after throwing an instance of 'Xapian::!DatabaseModifiedError'
(...)
set_overwritten: from block_to_cursor 10199 > 10197
terminate called after throwing an instance of 'Xapian::!DatabaseModifiedError'
set_overwritten: from block_to_cursor 10199 > 10197
terminate called after throwing an instance of 'Xapian::!DatabaseModifiedError'

I originally tested this with xapian-1.0.6, but it also occurs in 1.0.7.

I run xapian on Ubuntu Linux 8.04 (Hardy) with a 2.6.24-19-server kernel and an ext 3 file filesystem. The machine is an IBM x3650 with 40 GB RAM, and a ServeRAID-8k Controller running a Raid 10 over 6 SAS-Disks.

My most frequented index (the one that drops the exceptions) contains about 850.000 documents, needs 11 Gb of disk space, gets 5-15 updates per second, and about 20-25 search hits per second. I flush() this index every 10 minutes (which takes about 60-100 seconds + 4 seconds workaround delay ;-)

Some common punctuation (notably -) is treated as a word break when indexing, and as a phrase generator when searching. The problem is that many common cases end up creating phrase searches with one or two character terms which are very common, and these search are slow with a big database.

Examples include: {e-mail cd-r d-i-y

This could perhaps be addressed by a smarter word identifying algorithm. When indexing and searching, we could decide never to generate a single character term in certain circumstances (maybe also apply the same rules for two character terms).

So "e-mail" would be indexed as "email" not "e" and "mail". And similarly for searching. In general the extra conflation this gives seems useful (although email is apparently dutch for enamel...)

The query parser probably wouldn't apply this rule to quoted phrase searches - otherwise searching for "o freddled gruntbuggly" would search for "ofreddled gruntbuggly" and tragically not find any matches (I'm sure there are less esoteric examples - a search for "i robot" say...)

If we could somehow reduce the memory used by the postlist changes chert buffers, we could buffer more and/or let the OS have more spare memory for buffering disk blocks. That should allow indexing to run faster. However we need to compress in such a way that we can still implement Xapian::Database methods including the effects of the buffered changes.

Hello,

I locally first modified omindex to use libmagic's MIME database, instead of hard coding the MIME type to file extension mapping. This ensures that the internally used MIME types are more consistent with accepted standard types.

Then I went further and instead of using file extensions to determine type, used libmagic to fingerprint the files. This is slower, but ensures that the file actually is identified correctly even if the extension is wrong.

Now I am using libextractor to actually extract the metadata from the file, instead of calling these external programs inside omindex based on the MIME type. Using libextractor greatly simplifies omindex.

Is anyone interested in these modifications?

This bug is intended to just gather together enhancements we'd like to make to our Unicode support.

Currently I'm aware of two:

* Special cases for case conversion: http://www.unicode.org/Public/5.0.0/ucd/UCD.html#Case_Mappings and in particular: http://www.unicode.org/Public/5.0.0/ucd/SpecialCasing.txt

* Normalisation (mostly combining accents): http://www.unicode.org/Public/5.0.0/ucd/UCD.html#Decompositions_and_Normalization

I'd imagine we would probably want to target most such changes at 1.1.0, for reasons of database compatibility. There are probably cases where it would be reasonable to implement such changes sooner though - if we build a different database in a case where the existing behaviour is poor, or the difference isn't problematic for some other reason, say.

Now that we store both the stemmed and unstemmed forms of each word in the database, it might be nice to add a new stemming mode to the query parser which takes each word in the query and generates an "OR" query for it, with two parts; one being the unstemmed form and one being the stemmed form. This would mean that each query would match any document with words which match the stemmed form, but would give documents with the unstemmed form a higher weight.

We might call this option "STEM_BOTH", or some better name that someone other than me can think of.

Hi,

the patch at http://people.debian.org/~tviehmann/list-search/xapian_omega_add_option_filter_defaultop.diff adds an option map to allow overriding the filter behaviour from OR to AND among the terms of a given prefix. For example, if first and last name are indexed with prefix A, I would add

$setmap{filter_defaultop,A,AND}

to the query template in order to be handle first, last, or first and last name entered into the appropriate fields.

Kind regards

Thomas

URL: http://people.debian.org/~tviehmann/list-search/xapian_omega_add_option_filter_defaultop.diff

Currently, Xapian::Query::Internal stores any "double" parameter value as a sortable_serialised string. There is a FIXME in the code for set_dbl_parameter() and get_dbl_parameter() (around line 976 of api/omqueryinternal.cc) saying: "FIXME: rework for 1.1.0". This hasn't been changed until now due to fear of breaking ABI compatibility.

Instead, we should store double parameters as doubles in Query::Internal.

While reorganising this, it might be worth making parameter storage a bit more general, and tidying it up. We currently have the following parameters stored in Query::Internal:

• op: The operation to perform
• subqs: A list of subqueries
• parameter: A "termcount" - used by NEAR and PHRASE to be the window size, used by ELITE_SET to be the number of terms, and used by RANGE to be the value number to apply the range to. For the last of these, a "termcount" type isn't really appropriate (though it is probably the same storage size as "valueno" at present, so it probably works correctly).
• tname: A string holding the term, for a leaf query. The start of the range, for a range query.
• str_parameter: The end of the range for a range query. The result of sortable_serialise() on the multiplier for OP_SCALE_WEIGHT queries.
• term_pos: The position of the term for leaf queries.
• wqf: The within query frequency, for leaf queries.
• external_source: The external source, for external source queries.

Two approaches seem plausible to me - firstly, we could define a union with the possible parameter types, and store the parameters in a list of these unions. Alternatively, we could subclass Query::Internal for each of the possible query types, and just store the appropriate parameters for each.

The latter approach seems cleaner to me, and more likely to be flexible for future expansion of the available query operators, but I've not thought about this much yet.

The standard build system (ie, configure, autotools, etc) should be able to detect and use MSVC on a windows system. This will require some unix support stuff to be installed, but MSys & Mingw should suffice (ie, a full cygwin installation shouldn't be needed).

Mark has made some progress in this direction, and Richard made a start at looking at it, so this bug is intended as a place to collaborate.

Olly says: "I don't know if libtool's support has bitrotted though. If it has, there's a (very poorly named) wrapper called "wgcc" which translated gcc options to msvc"

Some code to do this kind of tokenisation is now available at http://code.google.com/p/cjk-tokenizer/ which should probably be used as the basis for supporting this in Xapian.

Currently, Enquire::register_match_decider() simply stores the values passed to it in the internals of the Enquire object. These values never get used.

Either register_match_decider() should be removed, or (more probably) the values should be used in the remote match case to allow match deciders registered with the server to be used.

For now, I've added a note in the documentation comment that this method effectively does nothing.

I am reporting this on behalf of Adel Gadllah <adel.gadllah@…>, who is looking into packaging the bindings for Fedora 7.

The PHP license and the GPL aren't compatible but xapian-bindings links PHP licenced and GPL licensed code.

Quotes from the conversation on IRC with Fedora developers : "the problem i'm seeing is that xapian-bindings has bits of code that are GPLv2+ and PHP" "and it is merging them together into one .cc file and compiling _that_" "except, the GPLv2 and PHP are incompatible" "BOOM" "tell upstream that they can't compile PHP code with GPL* code"

We need this solved first before continuing with building the other bindings in Fedora.

Fabrice

Currently, The Xapian::Weight::Internal class (which is, as of last night, the class holding the statistics for the whole collection used by the weight objects) is not publically visible. This means that it would be impossible, for example, for a user to write a weighting class equivalent to, say, the BM25Weight class, using the public API, because the statistics aren't available.

After cleaning up the weighting calculation system, I believe the

Xapian::Weight::Internal class is now nearly clean enough that it could reasonably be made public, allowing custom weighting classes access to all the statistics currently available.

We might want to make the termfreq and reltermfreq members private, since they're likely to be accessed mainly through the accessor functions anyway. Also we might want to combine them into a single map with entries holding both the termfreq and the reltermfreq, since it's usual to want to access both the termfreq and the reltermfreq for a particular term at the same time.

Also, we might want to call the class Xapian::Stats, instead of Xapian::Weight::Internal, to reflect the Stats being part of the public API, but this would require an ABI change, so would have to wait for 1.1.0. (We could keep the API compatible by making Xapian::Weight::Internal a typedef for Xapian::Stats, I think; currently Stats (with no namespace) is a typedef for Xapian::Weight::Internal).

Currently, there is no automated test of the behaviour when the replication function is doing a full copy of a database which gets modified while the copy is in progress.

I've done a manual test of this, so I'm moderately confident it works right, but I can't work out how to do a reliable automated test of it... at least, not without hacking a big sleep (or even a condition) into the database copying code, to allow me to be sure of getting some modifications done in the middle of it.

Any suggestions appreciated.

External PostingSources? have at least two annoying limitations (don't work with remote databases, don't work well with multi databases).

The newly added ValueWeightPostingSource? simply reads a value slot, returns documents with a non empty value, and returns the weight obtained by applying sortable_unserialise to the slot. Therefore, it could be implemented instead by a query operator, which would be similar to the existing OP_VALUE_... operators. This would make the feature available with remote databases and multi databases.

There may be a cleaner alternative which we haven't thought of yet, too.

Marking this for 1.1.0, since ValueWeightPostingSource? isn't yet in the API for any release, and we should remove it before making a release if we're going to remove it at all.

When the synonym query operator, and synonym postlists, are implemented, the queryparser should build wildcard queries using the synonym operator instead of the OR operator.

By default, the query parser splits words at spaces and applies lower-casing, stemming, and other normalisation to generate terms.

I believe that it should be possible to override the query parser's default behaviour for fields with a given set of prefixs, such that the query parser will treat some terms as literal text, allowing any character to occur in the term (including spaces and quotes), and not applying stemming or other normalisation to the term.

My thinking is that this can be implemented by adding a third prefix type (which I've called "EXACT_TEXT" for want of a better name), which causes the query parser to put all the characters following the prefix until the next space or ')' into the term (like terms with a "BOOL_FILTER" prefix type). The terms so generated are then included in the query structure in the same way as "FREE_TEXT" terms - ie, they obey surrounding boolean operators, and '+' and '-' prefixes.

In order to allow spaces (and ')' characters) in the terms, the query parser should support basic backslash escaping for the contents of such fields.

I have a patch which implements this that I'll attach to this bug report shortly. The patch has a few test cases (but more are needed for such a new feature), and has I've not written any documentation for it yet.

I know that Sidnei needs this for something he's working on, and I'd be delighted if we managed to get this into 1.0 since I'm going to have to maintain it until it gets committed, but it needs thorough review before being committed and timescales for 1.0 may not allow this.

I have a setup where I would like to be able to perform index updates one one master database, and then replicate this database to multiple client machines for searching.

I've experimented with using an NFS setup for this, with the database kept local on the index server and mounted remotely on the search clients, hoping that the client machines would keep enough of the database cached that the network traffic would not slow down searches too much. However, this method doesn't work satisfactorily because the NFS protocol doesn't allow NFS clients to get information about file updates other than by polling the mtime of a file: therefore, whenever the index is updated, any cached pages from the database are discarded. This leads to many very slow searches.

For now, I'm looking at setting up a system to take snapshots of databases using filesystem features (eg, the snapshot functionality provided by ZFS) and then using xdelta to calculate the differences between the databases, transferring the differences manually, and then applying the differences to the database on the search machines.

However, this approach has two major drawbacks: firstly, it depends on filesystem specific features (to take filesystem snapshots - a standard file copy could be used, but this would have poor cache performance, which is exactly what we're trying to avoid). Secondly, it requires the whole database to be traversed on the index machine to calculate the binary diffs. This is undesirable because it imposes unnecessary load on the index machine.

Instead, I would like to have a hook into flint which writes out a list of the modified btree pages, so that these can then be distributed to the search servers. If this information was written to a log file, together with the points at which fsync were called, and with details of the changes made to the base files, this log file could be transferred to the search machines, and could be replayed there, with minimal work required there.