Ticket #11 (closed defect: released)

Opened 6 years ago

Last modified 3 years ago

Make stemming language configurable in omindex and scriptindex

Reported by: arjen Owned by: olly
Priority: high Milestone:
Component: Omega Version: 0.7.4
Severity: minor Keywords:
Cc: Blocked By:
Operating System: All Blocking:

Description

As posted to the mailinglost:

When I do the simplest possible version of this query, I get these results: 0.6.5

All 10 matches OmQuery?((nos:(pos=1) AND meet:(pos=2))) Term frequencies: nos: 7,291, meeting: 97

1-10 of about 7,387 matches OmQuery?((nos:(pos=1) OR meet:(pos=2))) Term frequencies: nos: 7,291, meeting: 97

0.6.4

1-10 of about 52 matches OmQuery?((nos:(pos=1) AND meeting:(pos=2))) Term frequencies: nos: 7,291, meeting: 3,101

1-10 of about 10,360 matches OmQuery?((nos:(pos=1) OR meeting:(pos=2))) Term frequencies: nos: 7,291, meeting: 3,101

This is, of course, on the same database. The problem is in the number of found documents for the term 'meeting' (which gets stemmed to 'meet').

The problem only occurs with a 0.6.5 omega, an 0.6.4 with an 0.6.5 xapian-backend appears to be ok.

Change History

Changed 6 years ago by olly

  • owner changed from james to olly
  • version changed from other to 0.6.5

Changed 6 years ago by olly

  • status changed from new to assigned
  • severity changed from blocker to major

The OmQuery::get_description() results show that 0.6.4 seems to stem "meeting" to "meeting", while 0.6.5 stems "meeting" to "meet". "meet" is correct judging by http://snowball.tartarus.org/ at least.

This might explain the results as the database in question was built with 0.6.4.

Not sure about the results for 0.6.4 omega and the 0.6.5 xapian-core - that should behave the same as 0.6.5 omega and xapian-core by this theory...

Will investigate further...

Changed 6 years ago by arjen

  • severity changed from major to enhancement

*cough* If I change the stemming-language to Dutch it works correctly... And if I change the 0.6.4's stemming to english, that one fails aswell.

Couldn't you have the ./configure script change the stemming-language (defaulting to "english") ?

Changed 6 years ago by olly

  • severity changed from enhancement to minor
  • summary changed from Documents are not found with omega 0.6.5, while 0.6.4 does find them to Make stemming language configurable

configure options don't fit well with pre-compiled binary packages - they force the packager to choose some settings, and unless they're just a matter of enabling functionality (e.g. database backends) that means the packager is making choices for the user.

But this really should be configurable without editing the source code, and ideally on a per-database basis (with a per-installation default for newly created databases).

Changed 5 years ago by olly

Now configurable for omega (in CVS version) by putting $set{stemmer,nl} near the start of the omegascript template. Not configurable for omindex or scriptindex yet.

Changed 5 years ago by olly

  • rep_platform changed from PC to All
  • version changed from 0.6.5 to 0.7.4
  • op_sys changed from Linux to All
  • summary changed from Make stemming language configurable to Make stemming language configurable in omindex and scriptindex

Changed 3 years ago by olly

  • status changed from assigned to closed
  • resolution set to fixed

Now fixed in SVN trunk (omindex and scriptindex now take an optional --stemmer command line option).

Changed 3 years ago by olly

Fixed in release 0.9.3.

Changed 3 years ago by olly

  • resolution changed from fixed to released

Changed 3 years ago by trac

  • platform set to All
Note: See TracTickets for help on using tickets.