Ticket #59 (assigned enhancement)

Opened 4 years ago

Last modified 4 months ago

Compress chert postlist changes buffered in memory

Reported by: olly Owned by: olly
Priority: normal Milestone: 1.1.0
Component: Backend-Chert Version: SVN trunk
Severity: minor Keywords:
Cc: Blocked By:
Operating System: All Blocking:

Description (last modified by olly) (diff)

If we could somehow reduce the memory used by the postlist changes chert buffers, we could buffer more and/or let the OS have more spare memory for buffering disk blocks. That should allow indexing to run faster. However we need to compress in such a way that we can still implement Xapian::Database methods including the effects of the buffered changes.

Change History

Changed 4 years ago by olly

  • status changed from new to assigned

Changed 3 years ago by olly

  • summary changed from Compress quartz postlist changes buffered in memory to Compress flint postlist changes buffered in memory

This is really a candidate for flint, not quartz now.

We *can* fairly easily compress in the common (and speed critical) case of appending documents by simple storing a sorted list of entries for each term (probably in the same format we use on disk) as we're always appending to it.

If we keep the existing map, we can handle the pure modification case, and also a mixture of modifications and updates.

Changed 3 years ago by olly

  • rep_platform changed from Other to All
  • version changed from other to CVS HEAD
  • component changed from other to Backend-Flint
  • op_sys changed from other to All
  • severity changed from normal to enhancement

Changed 20 months ago by olly

  • priority changed from high to normal

Changed 15 months ago by olly

This is a patch which adds a quick hack implementation:

http://www.oligarchy.co.uk/xapian/patches/xapian-faster-flint-add-document.patch

It probably won't apply cleanly to SVN HEAD, it disables replace_document() and delete_document(), and it could store the appended changes in a much more compact way, but even this crude approach was measurably faster (I don't recall the exact figures, but something like 10-15% I think).

Changed 15 months ago by olly

  • blocking set to 120

This doesn't require API or ABI changes, so can go in 1.0.x.

Changed 15 months ago by trac

  • platform set to All

Changed 8 months ago by richard

  • description modified (diff)
  • milestone set to 1.1

Changed 8 months ago by richard

  • blocking deleted

(In #120) Remove the unfixed dependencies so we can close this bug - they're all marked for the 1.1.0 milestone.

Changed 4 months ago by olly

  • component changed from Backend-Flint to Backend-Chert
  • description modified (diff)
  • summary changed from Compress flint postlist changes buffered in memory to Compress chert postlist changes buffered in memory

Update to refer to chert rather than flint or quartz.

Note: See TracTickets for help on using tickets.