chert backend Segmentation Fault

[Tom Winch]

To reproduce on ogs21, say:

• Checkout trunk/Image and trunk/Shared

• Set PYTHONPATH to something like:

/home/maky/lib:/home/maky/ubuntu/usr/lib/python2.5/site-packages:../Shared

• from trunk/Image, execute:

rm -r /data/tests/imageref_db /data/tests/thumbnail_db python process_pages.py --chert --cache-size=1 /media/newcastle/disk/indexed-pages10.tar.bz2 /data/tests/imageref_db /data/tests/thumbnail_db

Output will be, e.g.

PID: 14796
ogs21: Processed (1: 8.5s, 0.00%, 65:04:09 remaining, 1741 thumbnails)
ogs21: Processed (2: 10.4s, 0.01%, 82:08:22 remaining, 3456 thumbnails)
ogs21: Processed (3: 9.9s, 0.01%, 74:40:47 remaining, 4680 thumbnails)
ogs21: Processed (4: 9.6s, 0.01%, 70:36:44 remaining, 6283 thumbnails)
ogs21: Processed (5: 12.7s, 0.02%, 95:53:52 remaining, 8127 thumbnails)
ogs21: Processed (6: 15.0s, 0.02%, 125:00:23 remaining, 9587 thumbnails)
ogs21: Processed (7: 9.0s, 0.03%, 56:14:48 remaining, 13156 thumbnails)
ogs21: Processed (8: 12.3s, 0.03%, 68:55:33 remaining, 16191 thumbnails)
ogs21: Processed (9: 14.2s, 0.03%, 125:08:20 remaining, 16878 thumbnails)
ogs21: Processed (10: 10.6s, 0.04%, 72:45:30 remaining, 19964 thumbnails)
ogs21: Processed (11: 11.7s, 0.04%, 87:00:38 remaining, 26966 thumbnails)
ogs21: Processed (12: 11.5s, 0.05%, 66:38:24 remaining, 29567 thumbnails)
ogs21: Processed (13: 11.8s, 0.05%, 73:56:15 remaining, 31904 thumbnails)
ogs21: Processed (14: 10.7s, 0.06%, 62:55:28 remaining, 33566 thumbnails)
ogs21: Processed (15: 10.4s, 0.06%, 83:37:02 remaining, 35146 thumbnails)
ogs21: Processed (16: 18.4s, 0.06%, 156:02:21 remaining, 36546 thumbnails)
Segmentation fault

Without the --chert flag, the flint backend is used and the process continues with no seg fault.

[Olly Betts]

Can you get a backtrace from where the SEGV happens?

[Tom Winch]

Traceback (most recent call last):
  File "process_pages.py", line 168, in <module>
    main(path)
  File "process_pages.py", line 124, in main
    process_file(archive_path, member.name, file)
  File "process_pages.py", line 80, in process_file
    if url: process_content(url, content)
  File "process_pages.py", line 66, in process_content
    image_db.insert(url, target, tuple[1:])
  File "/home/tomjon/trunk/Image/image_db.py", line 126, in insert
    _time(_insert, referer, target, values)
  File "/home/tomjon/trunk/Image/image_db.py", line 255, in _time
    return op(*args)
  File "/home/tomjon/trunk/Image/image_db.py", line 122, in _insert
    _thumbnail_fn(target, target_term, referer_term)
  File "/home/tomjon/trunk/Image/image_db.py", line 94, in _thumbnail_fn
    doc = mset[0].document
  File "/home/maky/ubuntu/usr/lib/python2.5/site-packages/xapian/__init__.py", line 1171, in _get_document
    self._document = self._mset._get_hit_internal(self.rank - self._firstitem).get_document()
DatabaseCorruptError: Expected block 0 to be level 1, not 0

*** glibc detected *** python: double free or corruption (!prev): 0x000000000084bf50 ***

[Richard Boulton]

I have a login to the machine that this error is occurring on, so I've assigned this bug to me, and will investigate it later today.

[Olly Betts]

FWIW, I meant a C++ backtrace, but the DatabaseCorruptError is interesting, as is the report of malloc heap corruption.

Anyway, as Richard says it's easier for him to investigate.

[Olly Betts]

Setting milestone to 1.1.0 - we really want to fix this before 1.1.0, and it's not relevant to 1.0.x unless it turns out to be a generic Btree bug which is only tickled by stuff chert is doing (or something like that).

Richard: Did you get a chance to look at this?

[Olly Betts]

Is this still a live issue, or did it turn out to be a problem elsewhere?

[Richard Boulton]

I can't reproduce this problem now, either with the version of Xapian installed on that machine, or with Xapian SVN trunk (though the version installed on the machine may have been updated since the problem occurred).

Marking it as WORKSFORME since I can't reproduce.