_xapian.so shoud link against libpythonX.X.so

[lilydjwg]

Currently it doesn't link against libpythonX.X.so and there are lots of Python symbols not defined as reported by ldd -r.

This does not matter when using with the Python executable, but it throws errors or even crashes inside a program embedding Python, e.g. Vim with +python/dyn feature.

[Richard Boulton]

Are you sure that linking to libpythonX.X.so is always the right thing to do? Is that library always even available (perhaps python might be linked statically on some platforms?)

Reading ​http://docs.python.org/2/extending/embedding.html#linking-requirements it sounds like linking to libpython isn't normally needed; might the problem here be a problem with the flags supplied to vim?

[lilydjwg]

I don't know about other platforms. I've checked that most of Python extensions on my system link to libpythonX.X.so, but not all.

I have this problem because my Vim is built with both Python 2 and 3 support. Those Python symbols are local (IIUC) so they won't conflict, which means Python extensions that do not links to libpythonX.X.so won't work.

Well, you can close this ticket; I will change LDFLAGS and build the xapian extension myself. But I hope the build script can link to the library when possible.

[Richard Boulton]

I wasn't actually advocating closing, I was trying to get more information to work out if this is something we should be changing. We don't want users to have to hack the build system to use Xapian...

The need to support python 2 and 3 is interesting; I think that would make it harder (or impossible) for vim to link the necessary symbols in itself, so its more reasonable that we should be linking to libpython ourself.

What platform are you on? We actually already link to libpython on windows.

[James Aylett]

Note that on Ubuntu 13.04 the python library should be linked both for py2 and py3:

user@ubuntu:~/nsfwcorp.com$ python3.3-config --ldflags
-L/usr/lib/python3.3/config-3.3m-x86_64-linux-gnu -L/usr/lib -lpthread -ldl  -lutil -lm  -lpython3.3m -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions
user@ubuntu:~/nsfwcorp.com$ python2.7-config --ldflags
-L/usr/lib/python2.7/config-x86_64-linux-gnu -L/usr/lib -lpthread -ldl  -lutil -lm  -lpython2.7 -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions

However the Ubuntu-shipped build does *not* link _xapian.so with libpythonX.X.so, suggesting something isn't quite right with the way we're picking up those flags in at least that case…?

[lilydjwg]

I'm on Arch Linux (64bit) and using a self-built Vim.

(My) Vim dlopens and dlsyms all the symbols from libpython* it needs. And on some platforms (e.g. Linux) it is possible to make symbols from two versions of Python not to conflict. AFAIK, Vim doesn't do anything special about how Python's C extensions link.

I need this change to use the completion function from pythoncomplete.vim shipped with Vim.

$ python3.3-config --ldflags
-lpthread -ldl -lutil -lm -lpython3.3m -Xlinker -export-dynamic
$ python2-config --ldflags
-lpthread -ldl -lutil -lm -lpython2.7 -Xlinker -export-dynamic

[Olly Betts]

The need to support python 2 and 3 is interesting; I think that would make it harder (or impossible) for vim to link the necessary symbols in itself, so its more reasonable that we should be linking to libpython ourself.

FWIW, that's probably not an inherent limitation - if vim passed RTLD_GLOBAL to dlopen() when it loaded whichever version of libpython the user wants to use, then the symbols from libpython would be available to anything that gets loaded by dlopen() afterwards. It probably couldn't support actually loading libpython2 and libpython3 at once (at least if they use some of the same symbols).

Anyway, we don't use python-config at all currently. I suspect it didn't even exist when the configure checks in xapian-bindings were written - looking at older debian packages, I see it in python-2.5 but not python-2.4. ​http://bugs.python.org/issue1161914 seems to be the ticket for adding it, but doesn't mention versions that I can see. It doesn't seem to be mentioned by any python docs apart from its own --help output and man page, so its existence isn't all that easy to discover.

So, python-config's docs suggest:

       To build a dynamically loadable python module, use

              gcc $(python-config --cflags --ldflags) -shared -fPIC progr.cpp -o progr.so

Which sounds good, but its --ldflags option conflates "linker arguments to use when embedding python into a program" with "linker arguments to use when linking a python module". I think distutils is a more reliable source for how such a module should be linked, and for the simple example from ​http://docs.python.org/2/extending/building.html on Debian wheezy I get (with demo.c just being an empty file):

$ python setup.py build_ext
running build_ext
building 'demo' extension
creating build
creating build/temp.linux-x86_64-2.7
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c demo.c -o build/temp.linux-x86_64-2.7/demo.o
creating build/lib.linux-x86_64-2.7
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-x86_64-2.7/demo.o -o build/lib.linux-x86_64-2.7/demo.so

Note that it doesn't link with libpython.

Interestingly, I checked the installed python modules on that machine and none of them are linked to libpython:

$ find /usr/lib/python2.7 -name '*.so' -print | wc -l
86
$ find /usr/lib/python2.7 -name '*.so' -print0 | xargs -0 -n1 ldd |grep -cw libz
34
$ find /usr/lib/python2.7 -name '*.so' -print0 | xargs -0 -n1 ldd |grep libpython
$

As richard says, on Windows we do link the compiled module with libpython - that's because a DLL can't have undefined symbols which get resolved by the application loading it (or some limitation like that). I don't have a windows machine to test on, but I'd expect distutils to link to libpython there.

I wasn't actually advocating closing, I was trying to get more information to work out if this is something we should be changing.

Yeah, it would definitely be good to sort out what's expected from extension modules here.

[Olly Betts]

The results of investigations I reported in comment:6 suggest we shouldn't be linking with libpython on most platforms. The fact that distutils (the standard build tool for python) doesn't link them seems a very strong argument.

We've had no feedback on my comment for 3 months, and keeping this ticket open indefinitely doesn't seem to serve a useful purpose, so I'm closing this as "notabug". If there are good reasons to think we really should be linking with these libraries, please comment and give them. But I suspect the issue is actually that vim should be calling dlopen with RTLD_GLOBAL.

[purple]

We hit similar problem with vim python/dyn bindings at Arch Linux. After googling I came to conclusion that modules *should* depend on python2.so at the systems where it present.

Not linking against pythhon2.so solves no issues and creates a lot of new problems that is not possible to resolve cleanly.

Upstream should not force "modules with dangled python2.so". Instead it should always link to python2.so if it presents at the system.

[Olly Betts]

Please can you provide links to the resources which led you to that conclusion?

As noted above, distutils seems to disagree with you.

[purple]

Please can you provide links to the resources which led you to that conclusion?

Google for "vim python dynamic RTLD_GLOBAL"

There are plenty of complains that Python extensions not properly linked to libpython.so cannot be loaded into vim. As an ugly workaround for such extensions it is suggested to load libpython.so into global namespace (RTLD_GLOBAL vim trick). But it crashes system if user tries to load both py2 and py3 extensions.