Consider using @rpath on OS X

[James Aylett]

OS X library linkage is by full path, but with an override using DYLD_LIBRARY_PATH (and friends) so that you can change the exact location of the library in use at runtime. However SIP (the system integrity protection) system on recent versions of OS X prevents DYLD_* environment variables from affecting "system" binaries, such as those in /usr/bin.

There's an alternative way of linking OS X dynamic libraries. By setting the library name to "@rpath/<leafname>" (eg install_name_tool -id '@rpath/libxapian.30.dylib' .libs/libxapian.30.dylib, or -install_name '@rpath/libxapian.30.dylib' to the linker), the RPATH segments of binary objects is used instead. This then has to be set (using -rpath to the linker command), and edited (using install_name_tool).

Either we could add both the build and install paths to all binaries built in-tree, or we could add the build version and then use install_name_tool on installation to change that to the install path. This would then allow us to drop all the DYLD_* use in -bindings, and should allow tests to run against system libraries (which is impossible at the moment because of DYLD_* pruning).

xapian-config --ldflags would then have to include -rpath <install-lib-path> so that clients can link with the installed version of Xapian. (Allow out-tree binaries to link with the in-tree uninstalled version seems unnecessary and prone to complication.)

Note that GNU libtool on OS X already sets -install_name, and there doesn't seem to be a way of overriding it.

[Olly Betts]

The OS X "libtool" has nothing to do with GNU libtool. Seems Apple just picked a name which GNU had been using for a well-known a project for years...

[Olly Betts]

Looking at the libtool source, it passes -install_name \$rpath/\$soname, though I'm not seeing exactly where rpath gets set. Can you attach the generated libtool script (i.e. xapian-core/libtool) from OS X?

[James Aylett]

OS X-generated GNU libtool (2.4.6)

[Olly Betts]

​http://apple.stackexchange.com/questions/215030/el-capitan-make-check-dyld-library-path suggests as a workaround we could just copy affected binaries to a subdirectory of the build tree and then set PATH to point there.

[Olly Betts]

That seems to help for Tcl, but not for other languages.

[Olly Betts]

We should try to push this to a resolution, probably one of:

• Use @rpath conditional on being on macOS
• Patch (GNU) libtool to handle this (and attempt to upstream the patch but we can bootstrap releases with a patched version at least)
• Document the limitation as proposed in #707:

  You cannot test against system version of languages (perl, tcl8, php, ruby, python2) with an uninstalled xapian-core, period. This will need noting somewhere, probably in the HACKING / developer guide docs.

[Olly Betts]

This has languished for too many years so I've documented the issue and how to workaround it in xapian-bindings/HACKING in f84c54fd4b3ec2d6df8b1c7d8fe56373b890e25f and am closing this as having been addressed. If the new text could be improved, please do so.

If someone developing on macos wants to improve the situation that would be great, but it's not something I can realistically do without a mac. I think it wouldn't be very hard to get xapian-core's top-level Makefile.am to do the equivalent of install_name_tool -id '@rpath/libxapian.30.dylib' .libs/libxapian.30.dylib - the trickiest part I can see is that we may need to do it conditional on there being an rpath having been set by libtool (because if there isn't an rpath, I suspect using @rpath may be problematic. This really needs someone with macos knowledge, or at least the ability to interactively test this.

Also we'd need to work out how best to hook in this extra command. Potentially it could be an explicit manual step to run if you want this to work, which at least would avoid risk of it breaking the user-install workflow.