*BSD conditional to modify xapian-bindings/config.ac to suppress -lstdc++ library

[David Gessel]

This is similar to #430, opened and marked invalid many years ago, and likewise the issue is, for me, arising from attempting to build a ruby gem xapian-ruby-1.4.22 on FreeBSD. The problematic code seems to be in xapian-bindings-1.4.22/configure.ac which generates a /configure file that includes at 21047

SWIG_CXXFLAGS=
if test yes = "$GXX" ; then
                    SWIG_CXXFLAGS="-fno-strict-aliasing"

              XAPIAN_LIBS="$XAPIAN_LIBS -lstdc++"


cat confdefs.h - <<_ACEOF >conftest.$ac_ext

The ​FreeBSD port for this gem, patches the same configure file as

--- configure.orig	2023-02-02 01:01:44 UTC
+++ configure
@@ -21048,7 +21048,7 @@ SWIG_CXXFLAGS=
 if test yes = "$GXX" ; then
                     SWIG_CXXFLAGS="-fno-strict-aliasing"
 
-              XAPIAN_LIBS="$XAPIAN_LIBS -lstdc++"
+              XAPIAN_LIBS="$XAPIAN_LIBS"
 
 
 cat confdefs.h - <<_ACEOF >conftest.$ac_ext

While this is easily patched manually after a build failure to get a successful build on FreeBSD, might it be possible to detect the build environment and modify the construction of the configure file from config.ac in a manner similar to ​the aitd project has?

AC_MSG_CHECKING(system)
case "$host_os" in
...
    mingw32* )
        LIBS="$XAPIAN_LIBS -lstdc++"
        ;;
...
    openbsd* )
...
        LIBS="$XAPIAN_LIBS"
        ;;

Adjusted as appropriate to include a FreeBSD detection.

[Olly Betts]

It's hard to know exactly what was going on in #430 as the interesting part of the log was not provided, but it was on "10.6.2 OS X" not FreeBSD anyway. From the information provided it looks to me more like a discrepancy in what different things are expecting for the module extension (.bundle vs .so) which we never saw in an upstream build so it seems likely it was a discrepancy between upstream xapian-bindings and the ruby gem. I think this annoying extension difference is an Apple invention, so not relevant for FreeBSD.

This explicit linking to -lstdc++ was actually added for OpenBSD (see the comment above it). It could probably be limited to just OpenBSD (I think we didn't originally in case other platforms might need it too, but it makes the assumption that the standard C++ library in use is libstdc++). If we're going to drop it for the BSDs it probably would be better to just drop it entirely.

It was added in 2005, and OpenBSD now seems to use clang as the default compiler, but I couldn't trivially find if the default C++ standard library is now libstdc++ or libc++ (clang supports both). ​https://man.openbsd.org/intro.3 mentions both of them for example.

If that default has changed, it might be we actually ought to explicitly link with -lc++ on OpenBSD (or perhaps better determine which C++ standard library is in use and explicitly link that).

It's also possible that the "deliberate decision on the part of OpenBSD developers" that required this has changed since and this isn't required on OpenBSD any more.

Unfortunately we don't have an OpenBSD CI job to easily test changes on. We're using ​https://github.com/vmactions for the non-Linux platform CI jobs and that has OpenBSD support so hopefully it's easy to add. I'll have a go.

[David Gessel]

Thanks! I haven't been able to get much clarity and certainly don't know enough myself to do more than test and report, which I'm more than happy to do. I don't think there is a hugely robust community running redmine on FreeBSD with xapian extensions via DMSF, oddly, and the nextcloud folks still haven't embraced it's Java free pleasures.

[Olly Betts]

I have a mostly there CI job for openbsd now - it gets far enough to show that openbsd doesn't want -lstdc++ nowadays.

My current thinking is to stop adding it, and document the workaround with a plea to let us know if it is needed. Ideally we'd test with GCC on openbsd too, but it doesn't seem to be packaged by openbsd ports so realistically is unlikely to be used much.

[David Gessel]

FreeBSD is normally built with LLVM from base and unless a build job is called through the standard process, it may not respect /etc/make.conf (where a specific compiler and options is typically specified) and then it's less deterministic which compiler gets called if GCC is installed in parallel. One might also install one of the supported LLVM versions from ports and that will generally be used rather than base. It can get complicated, but I think this will be the right solution until the next big compiler upheaval.

[Olly Betts]

I don't think FreeBSD ever needed -lstdc++ explicitly linking.

We hit this problem on OpenBSD nearly 20 years ago and found the solution of explicitly linking -lstdc++ :

​https://lists.xapian.org/pipermail/xapian-discuss/2005-September/001153.html

I think I worked out the underlying cause a little later but memory is hazy this far in the future. It must be something to do with dynamically loading a C++ shared object (xapian's Python bindings) into a program written in C (the Python interpreter). From the comment it seemed to be something the OpenBSD devs had chosen to do, but sadly I failed to add a reference to what that was. Something in their C/C++ toolchain presumably.

Rather than limit the change to just OpenBSD, it seemed reasonable to always add -lstdc++ when the compiler was identified as GCC because it was harmless when not needed, and might conceivably be needed on another platform which nobody had tested on. Testing for a platform by name is a potentially problematic approach - consider if somebody forked OpenBSD under a different name, we'd likely still want this (and that's not an entirely theoretical concern - e.g. dragonflybsd is a fork of FreeBSD). However new platforms are fairly rare and Xapian has been ported to most of the existing ones so in practice it's not such a bad approach.

This change was several years before clang arrived on the scene (wikipedia says 2007 for initial release, but 2010 or so for C++ support to be usable).

Somewhat unhelpfully but also somewhat understandably, clang deliberately impersonates GCC enough to trick autoconf into identifying it as GCC - they did this so it can be used to build existing projects without having to modify their build systems extensively. This wasn't a problem for our OpenBSD fix initially, but then libc++ came along (I didn't trivially find an exact year, but first mention in a xapian commit message was 2015; it becoming the default C++ library for clang was more recent though I think).

Now on FreeBSD, our configure identifies clang as GCC, so adds -lstdc++, but clang now uses libc++ by default and libstdc++ probably isn't even installed so the link fails (if libstdc++ was installed, the explicit link would probably be harmless, but may cause problems).

We do actually already go on to poke the compiler harder to see if it's is really GCC or clang (or intel's compiler which similarly tries to trick configure into thinking it is GCC) but it wouldn't be correct to move this addition to only be done for GCC because clang can be built to use libstdc++ (and it can be even be built to allow selection of libc++ or libstdc++ at runtime).

My recent testing on OpenBSD with the system clang shows that explicitly linking with libstdc++ fails (because libstdc++ isn't found) but not explicitly linking with libstdc++ also fails with:

   <internal:/usr/local/lib/ruby/3.3/rubygems/core_ext/kernel_require.rb>:136:in `require': Cannot load specified object - /root/xapian-bindings/ruby/.libs/_xapian.so (LoadError)
  	from <internal:/usr/local/lib/ruby/3.3/rubygems/core_ext/kernel_require.rb>:136:in `require'
  	from /root/xapian-bindings/ruby/xapian.rb:42:in `<module:Xapian>'
  	from /root/xapian-bindings/ruby/xapian.rb:40:in `<top (required)>'
  	from <internal:/usr/local/lib/ruby/3.3/rubygems/core_ext/kernel_require.rb>:136:in `require'
  	from <internal:/usr/local/lib/ruby/3.3/rubygems/core_ext/kernel_require.rb>:136:in `require'
  	from ./smoketest.rb:29:in `<main>'

This is not the same as the original error in that old email thread, but it could plausibly be the same cause (and modern Ruby returns a clean LoadError while Python, or at least 20-year old Python, allowed a splurge of dynamic linker errors).

Linking explicitly with -lc++ or -lc++abi doesn't seem to fix this. I haven't tried with Python instead of Ruby yet (now I've dug out the original email thread I see the original report was with Python).

[Olly Betts]

Perl bindings fail to load on OpenBSD:

#     Error:  Can't load '/root/xapian-bindings/perl/auto/Xapian/Xapian.so' for module Xapian: Cannot load specified object at /usr/libdata/perl5/amd64-openbsd/DynaLoader.pm line 206.

Python too:

/usr/local/bin/python3 -O -c 'import os;os.chdir("xapian");import _xapian'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: Cannot load specified object

So it seems none of the bindings actually work on OpenBSD in my CI job anyway.

OpenBSD seems to package the Perl, Python and RUby bindings, but just patch out the addition of -lstdc++:

​https://github.com/openbsd/ports/blob/master/databases/xapian-bindings/patches/patch-configure_ac

They don't seem to be doing anything special I can see:

​https://github.com/openbsd/ports/blob/master/databases/xapian-bindings/Makefile

The sed expression they apply doesn't look like it does anything to current 1.4.x - it was added 6 years ago to "Allow building with ruby 2.5"; not sure what the -fdeclspec stuff is about - it seems it was added for Ruby but I didn't get any issues with __declspec not being available so I think that's probably out of date too...

(I do wish packagers would all feed patches upstream - it'd make everyone's lives simpler.)

The vmactions stuff I'm using for CI runs everything as root inside a VM so I guess it could be a side effect of running as root.

Anyway, it seems clear we should just remove the -lstdc++ adding code completely.

[Olly Betts]

Removed for git master by 3f5ea70c4375ada31e35e553146d05dc1975a6ce. Needs backporting.

[Olly Betts]

Backported for 1.4.27 as 4bc74947dd4a20d099d80c71ace09392b57d0e56.

[David Gessel]

When ​Xapian-Ruby is patched, I'll test build and report back. Thanks for the fix. I hope I'm not the only person effected.