libxapian_jni.jnilib installed to preposterous location

[Ryan Carsten Schmidt]

Hi, I've just taken over maintenance of xapian in MacPorts, and I've noticed that the Java portion of xapian-bindings installs the file libxapian_jni.jnilib into a preposterous location:

$ port contents xapian-bindings-java
Port xapian-bindings-java contains:
  /opt/local/share/doc/xapian-bindings/java/examples/SimpleIndex.java
  /opt/local/share/doc/xapian-bindings/java/examples/SimpleSearch.java
  /opt/local/share/doc/xapian-bindings/java/index.html
  /opt/local/var/macports/build/_Users_rschmidt_macports_macports-ports-svn-trunk-new_devel_xapian-core/xapian-bindings-java/work/xapian-bindings-1.4.9/java/built/libxapian_jni.jnilib

MacPorts extracts and builds software in a temporary work directory which is deleted after installation. Note that libxapian_jni.jnilib was "installed" under the temporary work directory.

This is certainly happening in part because the file java/Makefile.am says:

jnidir = `pwd`/built

Unfortunately I'm not familiar with Java. I don't know how a Java user would use this jnilib file or where it should actually be installed to.

[Olly Betts]

(BTW, there should also be a .jar installed - not sure if you quoted all the output there or just the start of it...)

Unfortunately I'm not familiar with Java. I don't know how a Java user would use this jnilib file or where it should actually be installed to.

That's essentially the problem here - we don't know where to install the Java bindings in a usefully portable way, nor how to query the appropriate paths from Java, so for now we install to that built subdirectory and document that the user should copy the two files to the appropriate place(s).

E.g. for a Debian package I know we should install /usr/lib/jni/libxapian_jni.so and /usr/share/java/xapian.jar but I doubt that $(libdir)/jni/ and $(datarootdir)/java/ are correct on a Mac - it's probably somewhere under /Library/Java.

[Ryan Carsten Schmidt]

Replying to olly:

(BTW, there should also be a .jar installed - not sure if you quoted all the output there or just the start of it...)

Hmm, no, what I showed above is the entirety of what was installed. There is a built/xapian.jar file that gets built by make, but it doesn't get installed anywhere by make install.

Unfortunately I'm not familiar with Java. I don't know how a Java user would use this jnilib file or where it should actually be installed to.

That's essentially the problem here - we don't know where to install the Java bindings in a usefully portable way, nor how to query the appropriate paths from Java, so for now we install to that built subdirectory and document that the user should copy the two files to the appropriate place(s).

E.g. for a Debian package I know we should install /usr/lib/jni/libxapian_jni.so and /usr/share/java/xapian.jar but I doubt that $(libdir)/jni/ and $(datarootdir)/java/ are correct on a Mac - it's probably somewhere under /Library/Java.

Ah I see now that you do mention this in your java/index.html file:

on Mac OS X you can copy [the JNI library] into /Library/Java/Extensions/

I can certainly try putting the jnilib file there. Is that also the correct place to put the jar file or does that go somewhere else? I'll see if I can get your examples to work with that.

[Olly Betts]

on Mac OS X you can copy [the JNI library] into /Library/Java/Extensions/

I can certainly try putting the jnilib file there. Is that also the correct place to put the jar file or does that go somewhere else? I'll see if I can get your examples to work with that.

I'm not sure about the .jar.

It's possible to query java for these locations, but sadly the results don't seem to be suitable for automatically determining where to install these files. However they may provide insight:

$ cat tmptest.java 
public class tmptest {
    public static void main(String[] args) {
	System.out.println(System.getProperty("java.class.path"));
	System.out.println(System.getProperty("java.library.path"));
    }
}
$ javac tmptest.java 
$ java tmptest
.
/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib

The output above is for Debian unstable where the class path appears to be just the current directory (which doesn't seem right). The library path is more helpful, but starts with a directory which doesn't actually exist.

[Olly Betts]

I can certainly try putting the jnilib file there. Is that also the correct place to put the jar file or does that go somewhere else? I'll see if I can get your examples to work with that.

I'm not sure about the .jar.

I found the answer to this part at least - both the .jar and JNI library should go there:

Java can be extended by adding custom.jar, .zip, and .class files, as well as native JNI libraries, into an extensions directory. On some platforms this is designated by the java.ext.dir system property. In OS X, put your extensions in /Library/Java/Extensions/.

from ​https://developer.apple.com/library/archive/documentation/Java/Conceptual/Java14Development/01-JavaOverview/JavaOverview.html#//apple_ref/doc/uid/TP40001883-SW1

(I've mentioned this in our documentation in e6d1bb6d80dd751c797151a4e33690d7b6466e90.)

The note there about java.ext.dir also sounds promising, but this property seems to just be null for me on Linux (checked using the tmptest.java program above with the obvious extra line added). Search for that online suggests there's a java.ext.dirs property but that's also null for me.

Sadly I still haven't found a way to actually determine automatically where to install these files to, but hopefully the above helps with your packaging if you didn't already resolve this.

[Olly Betts]

I tested using tmptest.java on the GCC compile farm macOS machine (which is apparently 12.6 and an M1 CPU in case that matters), and java.library.path is /Users/olly/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:. while java.ext.dir and java.ext.dirs are both null.

It seems the first entry (/Users/olly/Library/Java/Extensions) doesn't exist by default though (neither does /Users/olly/Library/Java but /Users/olly/Library does exist). This would probably be an OK location as a default, but isn't where you'd want to install to as a packager anyway. However the first entry on Debian is still /usr/java/packages/lib which doesn't exist, and seems a bad default. I don't see a good way to tell if we should use the first entry on the path or not (e.g. directory existing or not doesn't work).

The full java.library.path on Debian unstable is currently /usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib. The second entry is probably the best place to install to (that's what the libxapian-jni uses). However the .jar should go elsewhere, and I don't see a way to query that location (libxapian-java puts it in /usr/share/java).

It really does seem it's just not possible to usefully query the appropriate directories from a Java install in a portable way, which seems stupid but sometimes the truth just is stupid. Given that, we aren't going to be able to ever full resolve this (unless Java itself addresses this) and it doesn't seem useful to keep it open indefinitely so I'm going to close it. We've documented this better at least. If anyone can see how to improve the situation, please speak up.