Opened 13 years ago

Closed 13 years ago

#587 closed enhancement (fixed)

[patch] Add option to limit maximum size of file indexed by omindex

Reported by: ndaley Owned by: Olly Betts
Priority: normal Milestone: 1.2.10
Component: Omega Version: 1.2.8
Severity: normal Keywords:
Cc: Blocked By:
Blocking: Operating System: All

Description

I've found that indexing very large files (80Mb+ csv files in my case) can cause the indexing process to be terminated, possibly due to excessive memory usage. I have added a command line switch (--max-size) to limit the maximum size of files to be indexed. Any file larger than this is skipped.

I have attached a patch against xapian-omega-1.2.8.

Attachments (1)

omindex.patch​ (1.9 KB ) - added by ndaley 13 years ago.
Patch to add --max-size switch to omindex

Download all attachments as: .zip

Change History (6)

by ndaley, 13 years ago

Attachment: omindex.patch​ added

Patch to add --max-size switch to omindex

comment:1 by Olly Betts, 13 years ago

Milestone: → 1.3.0
Status: new → assigned

Thanks. Marking to 1.3.0.

comment:2 by Olly Betts, 13 years ago

Are you happy with the patch licensing requirements:

​http://trac.xapian.org/browser/trunk/xapian-core/HACKING#L1214

comment:3 by ndaley, 13 years ago

I can confirm that I'm fine with both the GPL and MIT/x licenses, and my employer, Neoworks, has no copyright or moral covenant over the code.

comment:4 by Olly Betts, 13 years ago

Milestone: 1.3.1 → 1.2.10

Applied to trunk in r16450, with a few tweaks. The main one is you can say things like --max-size=10M or --max-size=0.25G rather than forcing the user to work the limit out in bytes.

This seems worth backporting, so marking for 1.2.10.

comment:5 by Olly Betts, 13 years ago

Resolution: → fixed
Status: assigned → closed

Backported for 1.2.10 in r16518.

Note: See TracTickets for help on using tickets.