chert backend Segmentation Fault

[tomjon]

To reproduce on ogs21, say:

- Checkout trunk/Image and trunk/Shared

- Set PYTHONPATH to something like:

/home/maky/lib:/home/maky/ubuntu/usr/lib/python2.5/site-packages:../Shared

- from trunk/Image, execute:

rm -r /data/tests/imageref_db /data/tests/thumbnail_db python process_pages.py --chert --cache-size=1 /media/newcastle/disk/indexed-pages10.tar.bz2 /data/tests/imageref_db /data/tests/thumbnail_db

Output will be, e.g.

PID: 14796
ogs21: Processed (1: 8.5s, 0.00%, 65:04:09 remaining, 1741 thumbnails)
ogs21: Processed (2: 10.4s, 0.01%, 82:08:22 remaining, 3456 thumbnails)
ogs21: Processed (3: 9.9s, 0.01%, 74:40:47 remaining, 4680 thumbnails)
ogs21: Processed (4: 9.6s, 0.01%, 70:36:44 remaining, 6283 thumbnails)
ogs21: Processed (5: 12.7s, 0.02%, 95:53:52 remaining, 8127 thumbnails)
ogs21: Processed (6: 15.0s, 0.02%, 125:00:23 remaining, 9587 thumbnails)
ogs21: Processed (7: 9.0s, 0.03%, 56:14:48 remaining, 13156 thumbnails)
ogs21: Processed (8: 12.3s, 0.03%, 68:55:33 remaining, 16191 thumbnails)
ogs21: Processed (9: 14.2s, 0.03%, 125:08:20 remaining, 16878 thumbnails)
ogs21: Processed (10: 10.6s, 0.04%, 72:45:30 remaining, 19964 thumbnails)
ogs21: Processed (11: 11.7s, 0.04%, 87:00:38 remaining, 26966 thumbnails)
ogs21: Processed (12: 11.5s, 0.05%, 66:38:24 remaining, 29567 thumbnails)
ogs21: Processed (13: 11.8s, 0.05%, 73:56:15 remaining, 31904 thumbnails)
ogs21: Processed (14: 10.7s, 0.06%, 62:55:28 remaining, 33566 thumbnails)
ogs21: Processed (15: 10.4s, 0.06%, 83:37:02 remaining, 35146 thumbnails)
ogs21: Processed (16: 18.4s, 0.06%, 156:02:21 remaining, 36546 thumbnails)
Segmentation fault

Without the --chert flag, the flint backend is used and the process continues with no seg fault.

[olly]

Can you get a backtrace from where the SEGV happens?

[tomjon]

Traceback (most recent call last):
  File "process_pages.py", line 168, in <module>
    main(path)
  File "process_pages.py", line 124, in main
    process_file(archive_path, member.name, file)
  File "process_pages.py", line 80, in process_file
    if url: process_content(url, content)
  File "process_pages.py", line 66, in process_content
    image_db.insert(url, target, tuple[1:])
  File "/home/tomjon/trunk/Image/image_db.py", line 126, in insert
    _time(_insert, referer, target, values)
  File "/home/tomjon/trunk/Image/image_db.py", line 255, in _time
    return op(*args)
  File "/home/tomjon/trunk/Image/image_db.py", line 122, in _insert
    _thumbnail_fn(target, target_term, referer_term)
  File "/home/tomjon/trunk/Image/image_db.py", line 94, in _thumbnail_fn
    doc = mset[0].document
  File "/home/maky/ubuntu/usr/lib/python2.5/site-packages/xapian/__init__.py", line 1171, in _get_document
    self._document = self._mset._get_hit_internal(self.rank - self._firstitem).get_document()
DatabaseCorruptError: Expected block 0 to be level 1, not 0

*** glibc detected *** python: double free or corruption (!prev): 0x000000000084bf50 ***

[richard]

I have a login to the machine that this error is occurring on, so I've assigned this bug to me, and will investigate it later today.

[olly]

FWIW, I meant a C++ backtrace, but the DatabaseCorruptError? is interesting, as is the report of malloc heap corruption.

Anyway, as Richard says it's easier for him to investigate.

[olly]

Setting milestone to 1.1.0 - we really want to fix this before 1.1.0, and it's not relevant to 1.0.x unless it turns out to be a generic Btree bug which is only tickled by stuff chert is doing (or something like that).

Richard: Did you get a chance to look at this?