Flint Record Table

This page describes the format of the Record table in the FlintBackend.

Key Format

pack_uint_preserving_sort(docid)

This takes one extra byte compared to Quartz, but means that when appending documents to a database, the insert is always in the same place (at the "end" of the table). This is faster, and produces a more compact database without a separate compaction step.

The plan for son-of-flint is to subclass the compare routine so we can store the key as compactly as Quartz does but keep the improved sort order.

Tag Format

The tag format is just the document data.

The tag may be stored compressed by zlib, which achieves ~30% compression on typical records of a few hundred characters. Very small tags (less than the minimum size of compressed output) are stored uncompressed.