​PyPI package of python bindings

[Abhilash Raj]

This seems to be a known issue which was previously discussed on -devel list[1] but seems like hasn't been picked up by anyone yet. There also seems to be a GSoC 2020 Ideas page [2], which seems to refer to this issue.

However, I couldn't find an actual issue in the bug tracker for this task, hence this ticket. I hope this is the step 1 to start conversation about what would it take to get it done and if I can potentially help out with the task.

[1]: ​https://lists.xapian.org/pipermail/xapian-devel/2013-January/001919.html

[2]: ​https://trac.xapian.org/wiki/GSoCProjectIdeas

[Olly Betts]

Hi,

You probably want to talk to jaylett as he's more familiar with python and pypi. I think he subscribes to ticket notifications, but you can also find him on IRC/matrix chat.

I notice there's

​https://pypi.org/project/xapian-bindings/

which looks plausible, though I've not tried it. That probably fulfils the role of "install xapian-core from a package, installing Xapian's python bindings via pypi. It may also be useful to have an alternative pypi package which bundles xapian-core (I don't really like such bundling myself, but it seems popular...)

[James Aylett]

From a quick look, xapian-bindings claims to detect -core then download the relevant -bindings release and installs from that. I still think a pypi package which doesn't have to download things is a good idea, which presents us with three options:

1. Bundle -core and -bindings against each versioned release.
2. Bundle only -bindings, and force people to match version with whatever -core they've installed.
3. Bundle -core and -bindings against each versioned release, but build & install -core only if there isn't one already installed which -bindings can work against.

1 seems to be the most useful. 3 sounds like a pain, and fragile (and hard to run comprehensive CI against). 2 has marginal advantages over the existing xapian-bindings in that you can download the package and install later without it having to download anything more; the downside is that you have to match it to your existing -core, so if that gets upgraded in the meantime you're stuck anyway.

So I think we should only consider 1, an official Xapian pypi package which contains -core and -bindings and so you can just do pip install xapian on any system we support. That could particularly be useful for minimal docker bases, where xapian-core is unlikely to be available by default.

There are probably three stages to this project, with value at each level:

1. Add make rules and the setuptools gubbins to build a working "any" wheel that contains the -core and -bindings source (including CI)
2. Register a pypi package and get this into our release process
3. Get binary wheels working, and perhaps use our existing CI infrastructure to build them for common platforms (so the package would contain a pre-built binary of -core and the python bindings library, plus the other bits of the package)

I've updated the GSoC project to point here.

[Kevin McCurley]

I was also looking for this, in part because the installation on macos installs the python bindings in a very peculiar place. The php bindings are similarly handicapped. The xapian-bindings python package has an explicit disclaimer on it, so perhaps that could be taken over by someone who actually maintains xapian.

[Olly Betts]

I can't speak for others but personally I don't really have the time to spare to take on maintaining a PyPI package, and I'm not an active Python user so not a good choice anyway.

We install both Python and PHP bindings based on what the language reports - for Python:

python3 -c 'import os,distutils.sysconfig;print(distutils.sysconfig.get_python_lib(1).replace(os.sep,"/"))'

(The separator replacement is just so it works on Microsoft Windows if a native Python which reports \ separated paths.)

If that doesn't report a helpful location on some platform then please help to improve that. A PyPI package is useful for people who want it, but isn't a really the appropriate fix for not working well out of the box on a particular platform.

Seems it is in need of an update anyway as distutils is now deprecated and with Python 3.10 the above emits:

<string>:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
<string>:1: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
/usr/lib/python3/dist-packages

For PHP:

php-config --extension-dir

That's documented as giving the "Directory where extensions are searched by default", so it seems strange if it really reports a "very peculiar place" on a mac, but again please help to improve that if necessary.

[Olly Betts]

I've since pushed a change to stop using distutils (as it's been deprecated for a while and is slated for removal in 3.12) and for Python3 we now use:

python3 -c 'import os,sysconfig;print(sysconfig.get_path("platlib").replace(os.sep,"/"))'

Does that report a more sensible path on macos?

If so then 1.4.20 will address this (working to get a release out soon).

If not, what path does this report?

And what path is reported by php-config --extension-dir?