Bindings build problem on OS X

[Tom]

There seems to be a bug in the current libtool which omits xapian-core/.libs from the search path when linking the bindings (python anyway). If there is no libxapian in /usr/... it will fail with a missing image error, otherwise it will attempt to link against it, which is probably a bad thing.

As a workaround, use DYLD_LIBRARY_PATH to force the linker, e.g.

DYLD_LIBRARY_PATH=/Users/maky/src/xapian/xapian-core/.libs make

[Olly Betts]

At least users building from tarballs are likely to build and install xapian-core before trying to build the bindings, so mostly this affects developers building from SVN.

But (if this hasn't been fixed by changes since) it would be good to resolve this. If it's a libtool issue, it'd be good to get a fix in the next release.

Tom: are you still seeing this?

[James Aylett]

I can confirm this is still an issue with trunk as of [13285]. It only affects the python bindings (php5, csharp, java-swig, ruby, tcl8 all work fine on my machine; I don't have the guile builddeps installed).

[Olly Betts]

We should try to fix this, but it only affects builds against uninstalled xapian-core, which is not a common case for end users, so not a 1.2.0 blocker.

[Olly Betts]

Hmm, looked at this and it's unclear to me what is going wrong.

James/Tom: Are you running make check or just make? If the later, then Python runs the module to create .pyc and .pyo files, whereas the other bindings only try to run it on make check, so that might explain the difference.

The web suggests OS X only gained rpath support fairly recently, so it might be libtool doesn't support it yet. Within itself, I think libtool can use DYLD_LIBRARY_PATH but it can't for this case as it isn't involved in running Python as things stand.

[Richard Boulton]

Potential, but unpleasant, solution

[Richard Boulton]

To answer the question in comment 4, the python bindings were failing on make, whereas the other bindings were failing only on make check.

The attached patch contains a possible solution to this; ie, set DYLD_LIBRARY_PATH directly, to point to the location that the xapian core library would be found when running an uninstalled SVN build. This seems rather horrible, but I implemented it for python to check the theory, and with the patch applied both "make" and "make check" succeed for OSX.

I tried to see if libtool has got anywhere with implementing this, but with no success. I think our options are either to apply a hack similar to that attached, or to upgrade to a version of libtool which fixes the problem (which may involve creating such a version of libtool!).

[Olly Betts]

Aside from the ugliness, there's one flaw I can see with the patch - just because ../../xapian-core/.libs exists doesn't mean that it's what we're currently trying to build against - I could check out an SVN tree, build it, make distclean in the bindings, and rebuild then against a different xapian-core.

That's clearly less bad than quietly using a library installed under /usr, but if DYLD_LIBRARY_PATH is meaningful on other platforms, we should narrow this hack to only take effect on Mac OS X. Looking at libtool.m4, libtool only tries to use DYLD_LIBRARY_PATH on darwin or rhapsody, so we're probably OK.

Even if we go with this hack for now, we should try to get libtool fixed too.

[Olly Betts]

Thinking about this some more, the issue here is caused by us going behind libtool's back, and copying _xapian.so from the .libs subdirectory.

Perhaps we ought to take the libtool-supported route to this (which is what we currently do for Java) and use something like this:

PYTHONPATH="xapian:$PYTHONPATH" ../libtool --mode=execute -dlopen _xapian.la python -c 'import _xapian'

Does that work on OS X?

Marking for 1.2.4 - if this approach works, we should apply it for the next release.

[Olly Betts]

Richard reported on IRC that the -dlopen didn't help find the right libxapian (from xapian-core).

I wonder why the other bindings work but Python doesn't...

[Richard Boulton]

The other bindings don't work - it's just that only the python bindings try running the generated bindings on "make" - all the bindings fail in the same way when their "make check" targets are run.

[Olly Betts]

r15151 sets DYLD_LIBRARY_PATH on OS X (only) to point into where an uninstalled libxapian would be in a combined tree from an SVN/git checkout. That's as good as we are likely to get right now, and the rest of this ticket is "suitable for 1.2.x".

[Olly Betts]

I've spotted a more sophisticated approach which should work for any uninstalled xapian-core, not just an SVN/git tree, and also won't try to build against ../xapian-core/.libs just because it exists.

Can someone with OS X test the attached patch please?

[Olly Betts]

Updated patch, with minor improvements.

[Olly Betts]

More sophisticated approach

[Olly Betts]

I realised I could do some rudimentary testing by faking $host and $host_os, and have fixed a few issues that uncovered. Updated patch attached.

[Olly Betts]

Dan offered to test this, so assigning to him.

Also marking for 1.3.1 (I'll backport if it's easy, though this issue mainly affects people building from SVN or git, so it matters most for trunk).

[Olly Betts]

Patch applied to trunk in r17032 - I've not been able to test it, but faking $host and $host_os allows reasonable testing, and it would be good to resolve this long-standing issue.

I think at this point in 1.2's release cycle, backporting this doesn't make much sense, as people aren't likely to be don't much development work on 1.2. If OS X developers feel differently, I'm happy to reconsider if someone can actually confirm the latest patch works as intended on OS X.

[Olly Betts]

This is useful for the buildbot, so backported to 1.2 branch in r17792.