python bindings do not build if core is built with --enable-shared=no

[Federico Schwindt]

Hi,

I've been using 0.9.10 and I've decide to upgrade to 1.0.2, but

xapian-bindings does not compile anymore if xapian-core is built with --enable-shared=no. I've also tried with --disable-shared.

The problem comes from the addition of compression (zlib) to the headers. The build stops with:

ImportError: /mypath/xapian-bindings-1.0.2/python/.libs/_xapian.so: undefined symbol: inflateInit2_

Cheers,

f.-

[Richard Boulton]

I don't get this error: instead, if I configure both core and bindings with --enable-shared=no, I get:

PYTHONPATH=".:.libs" /usr/bin/python -c "import _xapian;import xapian" Traceback (most recent call last):

File "<string>", line 1, in <module>

ImportError: No module named _xapian

(when running in the xapian-bindings module). Looking at the generated files, this is because _xapian.so hasn't been generated; which makes sense since it's a shared library, and we've asked xapian not to build such things.

Have you performed any workaround to get past this error? Alternatively it's possible you have a copy of _xapian.so lying around from an earlier compile (ie, before you specified --enable-shared=no), which has incompatible symbols. Try removing xapian-bindings/python/.libs/xapian.so and then recompiling, and report the output here.

[Federico Schwindt]

Hi,

Passing --enable-shared=no to the bindings is pointless, because as you

mention since it won't build the .so needed by python. What I want to do (and I've done it in the past with version 0.9.10) is to build _xapian.so with a static libxapian.a.

It's definetely not a problem in my environment and I can succesfully build

the python binding in this way using 0.9.10 but not with 1.0.2.

Cheers,

f.-

[Richard Boulton]

Aha! Now I've reproduced it - I get the behaviour you describe if I configure xapian-core with --enable-shared=no, but not xapian-bindings.

If I manually add "-lz" to the link line for _xapian.so, I get a working built python library. "-lz" is specified in the libxapian.la dependency line, so the problem is that for some reason the build with shared=no is not using the dependency libs information from libxapian.la correctly.

[Olly Betts]

Building the Python _xapian.so using a static libxapian.a isn't portable since it won't contain PIC code. It works on x86 (the code in the shared object will be modified to relocate it at the load address when it is loaded) but won't on x86_64 or indeed most other platforms. For example, on my box a xapian-bindings build against a static xapian-core fails with:

/usr/bin/ld: /home/olly/svn/xapian/xapian-core/.libs/libxapian.a(error.o): relocation R_X86_64_32 against `std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_S_empty_rep_storage' can not be used when making a shared object; recompile with -fPIC

I feel this is not a case that it is worth putting much effort into supporting, especially as there's an easy workaround (specify -lz manually). Since _xapian.so must be shared, we're talking about platforms which must support shared libraries anyway.

FWIW it looks like the problem is that libtool fails to pull in dependency_libs when you link a static library into a shared one, which looks like a bug in libtool. If you report it to the libtool developers and they fix it, this issue should be resolved when we upgrade to the libtool release with the fix in.

[Olly Betts]

It works on x86 (the code in the shared object will be modified to relocate it at the load address when it is loaded)

Hmm, I failed to explain why this is bad!

It means that any VM pages in _xapian.so (which contains all of xapian-core) which contain relocations (i.e. need to be modified for where the shared object gets loaded) can't be shared. If you only run one instance of a Xapian-using Python script then that doesn't make much difference (though a lot of relocations can mean it takes longer to load), but each extra python process will have its own private copy of a lot of pages rather than them all sharing a single copy as they would if you build xapian-core shared.

There's a downside to PIC code - usually you lose a CPU register (to hold a pointer to base address of the library, or something similar). But most Xapian applications tend to be limited by I/O rather than CPU, so that's less of a worry.

[Olly Betts]

I'm going to mark this as "WONTFIX". To summarise why: linking a non-PIC static library into a shared library doesn't work on most platforms, and has drawbacks on at least the most common platform where it does work, so I don't think we want to support such a configuration.

And if you really want to do this despite the drawbacks, there's an easy workaround in comment#3.

[Federico Schwindt]

Please correct me if I'm wrong, but you can generate a static library with objects compiled using -fpic or -fPIC (and thus containing PIC code) and then generate a shared object (_xapian.so) pulling the objects from it.

[Olly Betts]

I'm not aware of a way to easily build a "static with PIC code" library with libtool, but if you can persuade it to, this would probably work. There's a libtool option "--prefer-pic", but it doesn't seem to be documented beyond a terse comment in "libtool --help --mode=compile".