Java native libraries using JNA

Java and native code: most Java programmers have a strong dislike for it: it kills portability, it is messy and a lot of hassle. Then again: when you need it, you need it. Luckily JNA can make the experience less painful, if not outright enjoyable. In this article I will introduce JNA and companion tool jnaerator, as well as some tricks of my own involving object identity and loose coupling.

What is JNA?

The problem with native code is: having to build it. These days, installing most libraries is taken care of by your Linux distribution, MacPorts, or the developers distributing Windows DLLs. Installation is easy, but custom development is still hard: you have to have a C compiler at hand, figure out all kinds of flags and if you’re really unlucky also exotic build systems. When you use JNI, this is what you have to deal with.

Wouldn’t it be nice if all you had to do is install the resulting binary (meaning a .dll, .so or .dylib file) and use that directly from Java? That is exactly what JNA makes possible. It does away with compilation of bridging code using a C compiler. Instead, it generates this code on the fly. All you have to do is write a Java interface describing the C functions you would like to call, and JNA will hand you an implementation which actually does it. See the example on the JNA home page for details.

That leaves you with the boring job of going through the C API that you want to use and write your Java interface, including laying out C structures as Java classes.

An even easier way

Boring, mechanical work: those words sound like a challenge to a programmer. The jnaerator project steps up to the challenge: it is a tool which takes the C header files for a library and spits out the Java interface for use with JNA. Jnaerator also includes its own copy of JNA, so with just one download or Maven dependency, you’re all set.

It is worth noting that some common libraries, like the MacOSX frameworks and Mono/.NET have been converted and fine-tuned already and are available from the nativelibs4java project.

Jnaerator has different flavours of code it can generate. I opted for the new JNA Direct Mapping mode, which generates a Java class instead of an interface, with static methods which have the native modifier and no body, like this:

public static native int dInitODE2(int uiInitFlags);

This is the first function to call in a program which uses the ODE (Open Dynamics Engine), the library which I’m wrapping.

Creating a good Java API

Now that you have easy access to your C library, you are still not done. Because the generated glue code is inherently procedural in nature, it is up to you to layer some Object-Oriented magic on top.

Most of the time, the library will show signs of an OO design already, but using functions instead of methods. You will see a number of functions all taking a particular type as the first parameter. This type is often a pointer to some structure, which is used as a kind of handle for the objects that the library is modeling.

Here is a short example from ODE again:

public class OdeLibrary implements com.sun.jna.Library {
    // …
    public static native DWorld dWorldCreate();
    public static native void dWorldDestroy(DWorld world);
    public static native void dWorldSetGravity(DWorld world, double x, double y, double z);
    // …
}

Without delving into the specifics of ODE, what you see here are several static methods that do things with a DWorld object. If we want to make this more OO, we should move the methods related to DWorld to the DWorld class itself. Of course, we can’t actually move them, because these static native methods are the only ones that actually work, but we can add new methods to DWorld which call the static ones in OdeLibrary.
Luckily for us, jnaerator has already created the DWorld class, we just need to add some methods to it.

Here’s what DWorld looks like. It started out as a generated class with just two constructors, but I’ve added a static method and two object methods:

import static org.smop.odejna.OdeLibrary.*;
public class DWorld extends com.sun.jna.PointerType {
    // generated
    public DWorld(com.sun.jna.Pointer pointer) {
        super(pointer);
    }
    public DWorld() {
        super();
    }

    // hand coded
    public static DWorld create() {
        return dWorldCreate();
    }
    public void destroy() {
        dWorldDestroy(this);
    }
    public void setGravity(double x, double y, double z) {
        dWorldSetGravity(this, x, y, z);
    }
}

The static import makes all the functions available and it becomes very easy to write your methods, they mostly just forward to the functions.

How this works is: every opaque pointer type in C gets translated to a class extending PointerType. You are allowed to subclass these classes and change the signatures in the big JNA class to use them.

In ODE, some datatypes aren’t explicitly given their own C types, so they aren’t given distinct java types either. However, we can fix that.

Here’s an example, in OdeLibrary:

public static native DJoint dJointCreateBall(DWorld w, DJointGroup jg);
public static native DJoint dJointCreateHinge(DWorld w, DJointGroup jg);
public static native int dJointIsEnabled(DJoint j);
public static native void dJointSetBallAnchor(DJoint j, double x, double y, double z);
public static native void dJointSetHingeAnchor(DJoint j, double x, double y, double z);

It appears from these functions that there are operations which are applicable to every DJoint and then some that are specific to joints of specific types:

public static void main(String[] args) {
    // …
    DJoint ball = dJointCreateBall(world, null);
    int enabled = dJointIsEnabled(ball); // ok, applicable to any joint
    dJointSetBallAnchor(ball, 0.0, 0.0, 0.0); // ok, applicable to ball joints
    dJointSetHingeAnchor(ball, 0.0, 0.0, 0.0); // ERROR: C code will complain
}

We can introduce two subtypes of DJoint and have the types enforce the correct use of each method. JNA will still work if you replace some types with a subclass:

public class DBallJoint extends DJoint { }
public class DHingeJoint extends DJoint { }

public class OdeLibrary implements com.sun.jna.Library {
    // …
    public static native DBallJoint dJointCreateBall(DWorld w, DJointGroup jg);
    public static native DHingeJoint dJointCreateHinge(DWorld w, DJointGroup jg);
    public static native int dJointIsEnabled(DJoint j);
    public static native void dJointSetBallAnchor(DBallJoint j, double x, double y, double z);
    public static native void dJointSetHingeAnchor(DHingeJoint j, double x, double y, double z);
}

You would then make it more OO like so:

public class DJoint extends PointerType {
    // …
    public boolean isEnabled() {
        return dJointIsEnabled(this) != 0;
    }
}

public class DBallJoint extends DJoint {
    // …
    public static DBallJoint create(DWorld w, DJointGroup jg) {
        return dJointCreateBall(w, jg);
    }
    public void setAnchor(double x, double y, double z) {
        dJointSetBallAnchor(this, double x, double y, double z);
    }
}

public class DHingeJoint extends DJoint {
    // …
    public static DHingeJoint create(DWorld w, DJointGroup jg) {
        return dJointCreateHinge(w, jg);
    }
    public void setAnchor(double x, double y, double z) {
        dJointSetHingeAnchor(this, double x, double y, double z);
    }
}

public class Test {
    public static void main(String[] args) {
        // …
        DBallJoint ball = DBallJoint.create(world, null);
        boolean enabled = ball.isEnabled(); // applicable to any joint
        ball.setAnchor(0.0, 0.0, 0.0); // will call dJointSetBallAnchor()
    }
}

Two challenges

Object identity

After you’ve enthusiastically enhanced some classes, you hit a snag. You have added some of your own fields to a class and you find that sometimes the fields lose their values. What is going on?

For instance, if company is a wrapped C object and you call company.setDirector(dilbert) and then later call company.getDirector(), the object you get back is not the same as the one you put in. Instead, each time a C function returns one of your objects, you get a fresh one created using the default constructor and with just the pointer field set to the underlying C pointer.

If only we could remember our instances and make the library look them up when asked to return one…

Loose coupling

A second problem becomes apparent once you step back and look at your design from a distance: You wanted to create something beautiful, a wrapped library will have to be part of that, but it turns out to be a good thing in itself as well. Only: for your current project you just added some very specific fields and behavior to the wrapped classes, destroying any hope of being able to reuse the library. How can we extend the library in such a way that the extensions can be distributed separately from the generic wrapped library?

One solution

The key to the solution to both challenges is the

public Object fromNative(Object nativeValue, FromNativeContext ctx)

method in PointerType (the base class of our custom classes). The javadoc explains:

The default implementation simply creates a new instance of the class and assigns its pointer field. Override if you need different behavior, such as ensuring a single PointerType instance for each unique Pointer value, or instantiating a different PointerType subclass.

Aha! the creators of JNA foresaw our needs. In fact, the way I see it, these are such universal needs, there should be a universal way of satisfying them.

ExtendablePointerType

Just as easily said as done, I came up with a universal subclass of PointerType which your classes should inherit from instead. It will maintain a weak map of weak references so it can return instances that have been created earlier (you will have to prevent them from getting garbage collected yourself, the alternative was to pile them up indefinitely).
The second feature it has is that you can customize which classes it creates. Let’s say your current project needs some extra fields in the DBallJoint class but you don’t need them in the generic ODE library. What you can do is create a subclass MyBallJoint extends DBallJoint and then somewhere in your program register that when the API needs the one, it should instead use the other:

ExtendablePointerType.registerSubclass(DBallJoint.class, MyBallJoint.class);

The original static native methods will now return your subclass, but of course the signature hasn’t changed, so you will need to use casts. But as you move from procedural to OO style, it is easy to hide the casts inside the forwarding methods.

That is really all you need to know to use it, so to end this article, I’ll present the class in full.

Have fun hacking, go wrap some native code the easy way!

package org.smop.jna;

import java.lang.ref.WeakReference;
import java.util.HashMap;
import java.util.Map;
import java.util.WeakHashMap;

import com.sun.jna.FromNativeContext;
import com.sun.jna.Pointer;
import com.sun.jna.PointerType;

/**
 * Mapper from C to Java with custom subclasses and lasting identity.
 */
public class ExtendablePointerType extends PointerType {
    private static Map<Class<? extends PointerType>, Class<? extends PointerType>> subclassMap =
        new HashMap<Class<? extends PointerType>, Class<? extends PointerType>>();
    private static Map<Pointer, WeakReference<PointerType>> instances = new WeakHashMap<Pointer, WeakReference<PointerType>>();

    public static void registerSubclass(Class<? extends PointerType> clazz,
            Class<? extends PointerType> subclazz) {
        subclassMap.put(clazz, subclazz);
    }

    @Override
    public Object fromNative(Object nativeValue, FromNativeContext ctx) {
        // Always pass along null pointer values
        if (nativeValue == null) {
            return null;
        }
        PointerType res;

        // do we have an existing instance?
        WeakReference<PointerType> weakref = instances.get(nativeValue);
        if (weakref != null) {
            res = weakref.get();
            if (res != null)
                return res;
        }

        // determine target type
        @SuppressWarnings("unchecked")
        Class<? extends PointerType> targetType = ctx.getTargetType();
        if (subclassMap.containsKey(targetType))
            targetType = subclassMap.get(targetType);

        // Create new instance
        try {
            res = targetType.newInstance();
        } catch (InstantiationException e) {
            throw new IllegalArgumentException("Can't instantiate "
                    + ctx.getTargetType(), e);
        } catch (IllegalAccessException e) {
            throw new IllegalArgumentException("Not allowed to instantiate "
                    + ctx.getTargetType(), e);
        }
        res.setPointer((Pointer) nativeValue);

        // Store instance
        instances.put((Pointer) nativeValue,
                new WeakReference<PointerType>(res));

        return res;
    }
}