Error compiling Ruby bindings with Ruby from Nixpkg

[Graeme Porteous]

The Ruby symbols aren't available when compiling and results in an error:

========================================================================
      _wrap_LatLongCoords_size(int, unsigned long*, unsigned long) in xapian_wrap.o
      _wrap_Database_size(int, unsigned long*, unsigned long) in xapian_wrap.o
      _wrap_Database_get_total_length(int, unsigned long*, unsigned long) in xapian_wrap.o
      _wrap_Database_get_revision(int, unsigned long*, unsigned long) in xapian_wrap.o
      _wrap_Database_check(int, unsigned long*, unsigned long) in xapian_wrap.o
  "_rb_undef_alloc_func", referenced from:
      _Init__xapian in xapian_wrap.o
  "_rb_undef_method", referenced from:
      _Init__xapian in xapian_wrap.o
  "_rb_unexpected_type", referenced from:
      _Init__xapian in xapian_wrap.o
ld: symbol(s) not found for architecture arm64
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [Makefile:741: _xapian.la] Error 1
make[3]: Leaving directory './xapian-full/tmp/aarch64-apple-darwin24.2.0/ports/xapian-bindings/1.4.22/xapian-bindings-1.4.22/ruby'
make[2]: *** [Makefile:670: all] Error 2
make[2]: Leaving directory './xapian-full/tmp/aarch64-apple-darwin24.2.0/ports/xapian-bindings/1.4.22/xapian-bindings-1.4.22/ruby'
make[1]: *** [Makefile:514: all-recursive] Error 1
make[1]: Leaving directory './xapian-full/tmp/aarch64-apple-darwin24.2.0/ports/xapian-bindings/1.4.22/xapian-bindings-1.4.22'
make: *** [Makefile:436: all] Error 2
========================================================================

This can be solved by setting the LIBS environment variable to RbConfig::CONFIG["LIBRUBYARG"] as has been done here for our Ruby gem: ​https://github.com/mysociety/xapian-full/pull/11

I was wondering if a more appropriate change would be to update configure.ac to set RUBY_LIBS automatically if null, something along the lines of: ​https://github.com/mysociety/xapian-full/issues/10#issuecomment-2575244213

[Olly Betts]

I was wondering if a more appropriate change would be to update configure.ac to set RUBY_LIBS automatically if null, something along the lines of: ​​https://github.com/mysociety/xapian-full/issues/10#issuecomment-2575244213

The patch there is overly complicated since RUBY_LIBS is set to empty just before, so the amended conditional will always be true and it would be simpler and clearer to just scrap the conditionalisation entirely.

In the comment over there it says "Issue is with the compiler non being able to find the Ruby headers" but that's not correct as this is nothing to do with the headers - it's a link time problem to do with the symbols not being found in a library.

I don't understand why this fails though - most platforms allow undefined symbols when linking a shared object (which then get resolved by the dynamic linker when the library is loaded at runtime). The major exception is Microsoft Windows, essentially because DLLs are a really ancient technology and rather lacking in features, so there we pass -no-undefined to libtool in _xapian_la_LDFLAGS which means we need to explicitly link to libruby.

Can you show the full log mentioned in the output:

Running 'compile' for xapian-bindings 1.4.22... ERROR, review './xapian-full/tmp/aarch64-apple-darwin24.2.0/ports/xapian-bindings/1.4.22/compile.log' to see what happened. Last lines are:

We could enable linking to libruby everywhere but I'm not sure of the full implications - it seems like something we'd want to test carefully before changing which is a pain, especially for platforms we lack CI for.

[Olly Betts]

I wonder if you missed my question above...

Can you show the full log mentioned in the output:

Running 'compile' for xapian-bindings 1.4.22... ERROR, review './xapian-full/tmp/aarch64-apple-darwin24.2.0/ports/xapian-bindings/1.4.22/compile.log' to see what happened. Last lines are:

I'm guessing nixpkg is forcing linking with -z defs (or -Wl,-z,-defs or some equivalent or similar option) for some reason but it would be good to confirm.

[Graeme Porteous]

I have attached logs for configure and compile. I hope this helps understand what is going on here. I must admit I'm out of my depth understanding what is going on here.

Replying to Olly Betts:

I wonder if you missed my question above...

Can you show the full log mentioned in the output:

Running 'compile' for xapian-bindings 1.4.22... ERROR, review './xapian-full/tmp/aarch64-apple-darwin24.2.0/ports/xapian-bindings/1.4.22/compile.log' to see what happened. Last lines are:

I'm guessing nixpkg is forcing linking with -z defs (or -Wl,-z,-defs or some equivalent or similar option) for some reason but it would be good to confirm.

[Olly Betts]

Oh sorry, I missed this when preparing 1.4.28 (released earlier today) because I'd failed to set the milestone.

Thanks for the logs. They don't show use of any unusual linker options, so it isn't that.

Digging a bit, I found ​https://github.com/ziglang/zig/issues/8180 which seems to explain the situation - the one sentence summary is that macOS defaults to not allowing undefined symbols in this case, but you can select Linux-like behaviour with -undefined dynamic_lookup.

It looks like libtool should be turning this option on for us and I'm not sure why that isn't working for you. I can see that macOS-specific probes are getting run from your configure.log.

However looking at the handful of Debian-packaged Ruby modules I have installed, they all are explicitly linked to libruby3.3 except for _xapian.so, so it seems that linking to librubyX.Y unconditionally would better align us with Ruby norms so let's just do that.

Fix applied to git master in a3ef56cdda871cd2f6794ec50d686207a2032ae4 and backported for 1.4.29 in 6062f0031dd72998e848e7e3577886be1b6be2c7. This should be functionally the same as the patch you linked to above, but if you're able to easy test please do.

[Olly Betts]

I've had to make a rapid 1.4.29 release to fix incorrect library versioning in xapian-core 1.4.28.

The fix I applied (plus a tweak to try to pass a suitable -L option) is failing on macos, freebsd and dragonflybsd in CI so needs further work, so I've revert it to make the release.

Also reopening since this needs further work.

[Olly Betts]

Reapplied the patch and some additional tweaks and this is now passing all CI jobs. Should be in 1.4.30.