Build issue with 1.14.4 under python 3.12

[rouilj]

I am trying to compile 1.4.14 bindings for use with python 3.12 in my CI/CD pipeline for the Roundup Issue Tracker. We use xapian for one of our full text search implementations.

• OS: ubuntu 20.04
• gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
• binding source: ​https://oligarchy.co.uk/xapian/1.4.14/xapian-bindings-1.4.14.tar.xz
• xapian-core/libxapian-dev from ubuntu 20.04 apt repo

In addition to some deprecation warnings (that I think also occur when building under Python 3.11), we get syntax errors of the form:

In file included from /opt/python/3.12-dev/include/python3.12/Python.h:45,
                 from xapian_wrap.cc:22:
xapian_wrap.cc: In function ‘PyObject* SwigPyObject_New(void*, swig_type_info*, int)’:
/opt/python/3.12-dev/include/python3.12/objimpl.h:134:45: error: expected primary-expression before ‘)’ token
  134 | #define PyObject_New(type, typeobj) ((type *)_PyObject_New(typeobj))
      |                                             ^
/opt/python/3.12-dev/include/python3.12/objimpl.h:138:37: note: in expansion of macro ‘PyObject_New’
  138 | #define PyObject_NEW(type, typeobj) PyObject_New((type), (typeobj))
      |                                     ^~~~~~~~~~~~
xapian_wrap.cc:2148:24: note: in expansion of macro ‘PyObject_NEW’
 2148 |   SwigPyObject *sobj = PyObject_NEW(SwigPyObject, SwigPyObject_type());
      |                        ^~~~~~~~~~~~
/opt/python/3.12-dev/include/python3.12/objimpl.h:134:46: error: expected ‘)’ before ‘_PyObject_New’
  134 | #define PyObject_New(type, typeobj) ((type *)_PyObject_New(typeobj))
      |                                     ~        ^~~~~~~~~~~~~
/opt/python/3.12-dev/include/python3.12/objimpl.h:138:37: note: in expansion of macro ‘PyObject_New’
  138 | #define PyObject_NEW(type, typeobj) PyObject_New((type), (typeobj))
      |                                     ^~~~~~~~~~~~
xapian_wrap.cc:2148:24: note: in expansion of macro ‘PyObject_NEW’
 2148 |   SwigPyObject *sobj = PyObject_NEW(SwigPyObject, SwigPyObject_type());
      |                        ^~~~~~~~~~~~

The full log is visible at: ​https://app.travis-ci.com/github/roundup-tracker/roundup/jobs/574465489#L980. Before that line are the deprecations.

Is this build supposed to work the same as for Python 3.10, 3.11?

Thanks.

-- rouilj

[Olly Betts]

Why are you testing this with 1.4.14 when the latest release is 1.4.19?

Debian unstable doesn't have Python 3.12 yet, but with 3.11 the build fails because the shared object extension is detected as None:

python3.11 -c 'import distutils.sysconfig;print(distutils.sysconfig.get_config_vars("SO")[0])'

I think the fix for this and your problem is probably to address all the deprecation warnings about distutils (which if true has been removed in 3.12).

[rouilj]

Hi Olly:

Thanks for your feedback. I am using 1.4.14 because that is what is supplied with Ubuntu 20.04:

​https://ubuntu.pkgs.org/20.04/ubuntu-universe-amd64/libxapian30_1.4.14-2_amd64.deb.html

Ubuntu 20.04 is the newest build environment for travis ci and is the default for GitHub actions as well. Github was only able to release a beta of 22.04 Ubuntu last month.

My Python 3.11 build works fine because I modify the configure script to use:

  python3.11 -c 'import sysconfig;print(sysconfig.get_config_vars("EXT_SUFFIX")[0])'

which can be seen at:

​https://app.travis-ci.com/github/roundup-tracker/roundup/jobs/574465489#L785

with the value:

  PYTHON3_SO=`$PYTHON3 -c 'import sysconfig;print(sysconfig.get_config_vars("EXT_SUFFIX")[0])'`

The deprecation warnings I referred to aren't related (AFAICT) to distutils. They are g++ compiler warnings. For example:

xapian_wrap.cc: In constructor ‘XapianSWIG_Python_Thread_Block::XapianSWIG_Python_Thread_Block()’:
xapian_wrap.cc:87:32: warning: ‘int PyEval_ThreadsInitialized()’ is deprecated [-Wdeprecated-declarations]
   87 |  if (PyEval_ThreadsInitialized()) {
      |                                ^
...

Thoughts on why the macro expansion seems to be going wrong? Anything I can turn on to get more info for debugging.

[Olly Betts]

In practice mismatching xapian-bindings and xapian-core versions will often work together, provided they're from the same release series. There's a warning about it as sometimes it doesn't work and it's not feasible for us to test all such combinations (since there are O(n²) of them - nearly 400 for 1.4 currently) - the simple and reliable advice is to stick to matching versions, but if you're trying to test something like this it's definitely sensible to test with the latest version you can.

You can also get Ubuntu packages of a newer version of Xapian from ​https://launchpad.net/~xapian/+archive/ubuntu/backports - that's maintained by us (we take packages from Debian, just like Ubuntu themselves do, but we rebuild them for all current Ubuntu releases).

We're not going to be able to fix something in 1.4.14 retrospectively so any fix for this will be in a new stable release.

Also, if you're patching xapian-bindings but don't tell us, it's much harder to help you. It's also really helpful to forward such patches upstream.

[rouilj]

Hello Olly:

What leads you to believe I am mismatching xapian bindings and xapian core? I am building against core version 1.4.14 and I am building bindings version 1.4.14. Quite a bit of the compile script deals with getting exactly the binding version that matches the installed core (and libxapian-devel). Indeed with all version of python from 3.6 to 3.11 the compile completes just fine.

I have not figured out how to use any xapian-bindings installed via apt-get/dpkg. The Python environment I am running inside of is a Python virtual environment that has been created without xapian-bindings. Installing any package from a repo will not change the virtual python environment used for testing.

Is your suggestion to upgrade the xapian core/devel packages for my Ubuntu 3.12 and try recompiling a matching xapian-bindings? If so, does your continuous integration environment have a build of xapian that works for Python 3.12?

I should have mentioned that I fixed the configure script so it returns the proper shared object extension. However it looks like the build fails to compile far before that ever becomes an issue.

Did you look at the error message in context? If so does it indicate that a newer version of xapian-bindings might work with updates for swig and Python 3.12?

Thanks.

-- rouilj

[Olly Betts]

What leads you to believe I am mismatching xapian bindings and xapian core?

You have me backwards - it seems you're using version 1.4.14 xapian-bindings because the Ubuntu version you're using packages xapian-core 1.4.14, and I'm saying that you don't necessarily need to do that, and especially for debugging the problems you're hitting it'd be more helpful to test with the latest stable release as that will likely have fixes for at least some of the problems. Testing with the RELEASE/1.4 branch from git is arguably better still, but requires disproportionally more effort to close a much smaller window of potential fixes.

For example, xapian-bindings 1.4.19 should have fixed warning: ‘int PyEval_ThreadsInitialized()’ is deprecated, and earleir today I pushed fixes to the RELEASE/1.4 branch for SO to EXT_SUFFIX change, and for the deprecation warnings I still saw with Python 3.11.

Is your suggestion to upgrade the xapian core/devel packages for my Ubuntu 3.12 and try recompiling a matching xapian-bindings? If so, does your continuous integration environment have a build of xapian that works for Python 3.12?

I'm suggesting you test with the latest xapian-bindings you easily can. If that works with the 1.4.14 xapian-core packages then fine, but you might need to pull newer xapian-core packages from the PPA.

You won't be able to use packaged xapian-bindings with a newer Python version as the module currently needs recompiling for each Python minor version. Maybe one day we'll support the Python stable ABI but we don't currently.

Currently our CI just tests whatever the default python3 version packaged for each test environment is.

Did you look at the error message in context? If so does it indicate that a newer version of xapian-bindings might work with updates for swig and Python 3.12?

I did, but more context didn't help.

But as a general strategy, if you're trying to debug a compatibility issue like this then my advice would be to test with the latest version you can as it helps avoid wasting time and effort chasing down problems that have already been fixed.

I had a quick look at Python's git repo to see what that header currently is, and I think you've probably hit a Python bug as they merged a fix 15 hours ago to the code where the error is reported:

​https://github.com/python/cpython/pull/94234/files

However I see Python 3.12 isn't due to be released for over 15 months (at least according to ​https://peps.python.org/pep-0693/) and at this stage you probably need be prepared to deal with this sort of breakage yourself, or else wait to test with it until it's more stable.

[Olly Betts]

I'm convinced this was a bug in the in-development Python rather than Xapian; no feedback from the submitter to the contrary so closing.

[rouilj]

Just an update. Supposedly nightly got that patch and it still failed to build with the same issues.

However doing a change/commit to git/wait for it to fail, and eating credits all the while makes it a long expensive cycle.

Let's leave this closed for now. I'll keep running tests and when we get to the point where 3.12 hits beta and I have to start supporting it I may reopen this or create a new issue.

Thanks for your help so far.

-- rouilj

[Olly Betts]

You should be able to tell from the errors if you have the bug fix - from the report above:

/opt/python/3.12-dev/include/python3.12/objimpl.h:134:46: error: expected ‘)’ before ‘_PyObject_New’
  134 | #define PyObject_New(type, typeobj) ((type *)_PyObject_New(typeobj))
      |                                     ~        ^~~~~~~~~~~~~
/opt/python/3.12-dev/include/python3.12/objimpl.h:138:37: note: in expansion of macro ‘PyObject_New’
  138 | #define PyObject_NEW(type, typeobj) PyObject_New((type), (typeobj))
      |                                     ^~~~~~~~~~~~

The problem here is the ( and ) around type on line 138 - expanding the two macros shown here on the PyObject_NEW invocation shown from the Xapian code: PyObject_NEW(SwigPyObject, SwigPyObject_type()) -> PyObject_New((SwigPyObject), (SwigPyObject_type())) -> (((SwigPyObject) *)_PyObject_New((SwigPyObject_type()))) and that fails to compile because the ((SwigPyObject) *) cast is invalid C++ - it should be (SwigPyObject *).

If you still see (type) there in the error output you don't have the fix.

But anyway, yes - let's leave worry about Python 3.12 compatibility until it's nearer release.