Passing a collection of JNA structures to native method

Question

The Problem

I am attempting to pass a collection of JNA structures to a native method but it's proving very fiddly:

Let's say we have a structure:

class MyStructure extends Structure {
    // fields...
}

and a method in a JNA interface:

void pass(MyStructure[] data);

which maps to the native method:

void pass(const MYStructure* data);

Now the complication comes from the fact that the application is building a collection of these structures dynamically, i.e. we are NOT dealing with a static array but something like this:

class Builder {
    private final Collection<MyStructure> list = new ArrayList<>();

    // Add some data
    public void add(MyStructure entry) {
        list.add(entry);
    }

    // Pass the data to the native library
    public void pass() {
        // TODO
    }
}

A naive implementation of the pass() method could be:

MyStructure[] array = list.toArray(MyStucture[]::new);
api.pass(array);

(where lib is the JNA library interface).

Of course this doesn't work because the array is not a contiguous block of memory - fair enough.

Rubbish Solution #1

One solution is to allocate a JNA array from a structure instance and populate it field-by-field:

MYStructure[] array = (MyStructure[]) new MyStructure().toArray(size);
for(int n = 0; n < array.length; ++n) {
    array[n].field = list.get(n).field;
    // other fields...
}

This guarantees the array consist of contiguous memory. But we have had to implement a field-by-field copy of the data (which we've already populated in the list) - this is OK for a simple structure, but some of the data I am dealing with has dozens of fields, structures that point to further nested arrays, etc. Basically this approach is just not viable.

Rubbish Solution #2

Another alternative is to convert the collection of data to a simple JNA pointer, something along these lines:

MyStructure[] array = list.toArray(MyStructure[]::new);
int size = array[0].size();
Memory mem = new Memory(array.length * size);
for(int n = 0; n < array.length; ++n) {
    if(array[n] != null) {
        array[n].write();
        byte[] bytes = array[n].getPointer().getByteArray(0, size);
        mem.write(n * size, bytes, 0, bytes.length);
    }
}

This solution is generic so we can apply it to other structure as well. But we have to change the method signatures to be Pointer instead of MyStructure[] which makes the code more obtuse, less self-documenting and harder to test. Also we could be using a third-party library where this might not even be an option.

(Note I asked a similar question a while ago here but didn't get a satisfactory answer, thought I'd try again and I'll delete the old one / answer both).

Summary

Basically I was expecting/hoping to have something like this:

MyStructure[] array = MyStructure.magicContiguousMemoryBlock(list.toArray());

similar to how the JNA helper class provides StringArray for an array-of-string:

StringArray array = new StringArray(new String[]{...});

But no such 'magic' exists as far as I can tell. Is there another, simpler and more 'JNA' way of doing it? It seems really dumb (and probably incorrect) to have to allocate a byte-by-byte copy of the data that we essentially already have!

Do I have any other options? Any pointers (pun intended) gratefully accepted.

Answer 1

If you able to create a continues block of memory, why don't you simply de-serialize your list into it.

I.e. something like:

MyStructure[] array = list.get(0).toArray(list.size());
list.toArray(array);
pass(array);

In any case you'd better not to store Structure in your List or any another collection. It is better idea to hold a POJO inside, and then remap it to array of structures directly using a bean mapping library or manually.

With MapStruct bean mapping library it may looks like:

@Mapper
public interface FooStructMapper {
    FooStructMapper INSTANCE = Mappers.getMapper( FooStructMapper.class );
    void update(FooBean src, @MappingTarget MyStruct dst);
}

MyStrucure[] block = new MyStructure().toArray(list.size());
for(int i=0; i < block.length; i++) {
   FooStructMapper.INSTANCE.update(list.get(i), block[i]);
}

What the point - Structure constructor allocates memory block using Memory, it is really slow operation. As well as memory allocated outside of java heap space. It is always better to avoid this allocate whenever you can.

Answer 2

As the author of the previous answer, I realize a lot of the confusion was approaching it one way before realizing a better solution that we discussed primarily in comments to your answer. I will try to answer this additional clarification with an actual demonstration of my suggestion on that answer which I think is the best approach. Simply, if you have a non-contiguous structure and need a contiguous structure, you must either bring the contiguous memory to the structure, or copy the structure to the contiguous memory. I'll outline both approaches below.

Is there another, simpler and more 'JNA' way of doing it? It seems really dumb (and probably incorrect) to have to allocate a byte-by-byte copy of the data that we essentially already have!

I did mention in my answer on the other question that you could use useMemory() in this situation. It is a protected method but if you are already extending a Structure you have access to that method from the subclass (your structure), in much the same way (and for precisely the same purpose) as you would extend the Pointer constructor of a subclass.

You could therefore take an existing structure in your collection and change its native backing memory to be the contiguous memory. Here is a working example:

public class Test {

    @FieldOrder({ "a", "b" })
    public static class Foo extends Structure {
        public int a;
        public int b;

        // You can either override or create a separate helper method
        @Override
        public void useMemory(Pointer m) {
            super.useMemory(m);
        }
    }

    public static void main(String[] args) {
        List<Foo> list = new ArrayList<>();
        for (int i = 1; i < 6; i += 2) {
            Foo x = new Foo();
            x.a = i;
            x.b = i + 1;
            list.add(x);
        }

        Foo[] array = (Foo[]) list.get(0).toArray(list.size());
        // Index 0 copied on toArray()
        System.out.println(array[0].toString());
        // but we still need to change backing memory for it to the copy
        list.get(0).useMemory(array[0].getPointer());
        // iterate to change backing and write the rest
        for (int i = 1; i < array.length; i++) {
            list.get(i).useMemory(array[i].getPointer());
            list.get(i).write();
            // Since sending the structure array as an argument will auto-write,
            // it's necessary to sync it here.
            array[1].read(); 
        }
        // At this point you could send the contiguous structure array to native.
        // Both list.get(n) and array[n] point to the same memory, for example:
        System.out.println(list.get(1).toString());
        System.out.println(array[1].toString());
    }

Output (note the contiguous allocation). The second two outputs are the same, from either the list or the array.

Test$Foo(allocated@0x7fb687f0d550 (8 bytes) (shared from auto-allocated@0x7fb687f0d550 (24 bytes))) {
  int a@0x0=0x0001
  int b@0x4=0x0002
}
Test$Foo(allocated@0x7fb687f0d558 (8 bytes) (shared from allocated@0x7fb687f0d558 (8 bytes) (shared from allocated@0x7fb687f0d558 (8 bytes) (shared from allocated@0x7fb687f0d550 (8 bytes) (shared from auto-allocated@0x7fb687f0d550 (24 bytes)))))) {
  int a@0x0=0x0003
  int b@0x4=0x0004
}
Test$Foo(allocated@0x7fb687f0d558 (8 bytes) (shared from allocated@0x7fb687f0d558 (8 bytes) (shared from allocated@0x7fb687f0d550 (8 bytes) (shared from auto-allocated@0x7fb687f0d550 (24 bytes))))) {
  int a@0x0=0x0003
  int b@0x4=0x0004
}

If you don't want to put useMemory in every one of your structure definitions you can still put it in an intermediate class that extends Structure and then extend that intermediate class instead of Structure.

---

If you don't want to override useMemory() in your structure definitions (or a superclass of them), you can still do it "simply" in code with a little bit of inefficiency by copying over the memory.

In order to "get" that memory to write it elsewhere, you have to either read it from the Java-side memory (via reflection, which is what JNA does to convert the structure to the native memory block), or read it from Native-side memory (which requires writing it there, even if all you want to do is read it). Under-the-hood, JNA is writing the native bytes field-by-field, all hidden under a simple write() call in the API.

Your "Rubbish Solution #2" seems close to what's desired in this case. Here are the constraints that we have to deal with, with whatever solution:

• In the existing list or array of Structure, the native memory is not contiguous (unless you pre-allocate contiguous memory yourself, and use that memory in a controlled manner, or override useMemory() as demonstrated above), and the size is variable.
• The native function taking an array argument expects a block of contiguous memory.

Here are the "JNA ways" of dealing with structures and memory:

• Structures have native-allocated memory at a pointer value accessible via Structure.getPointer() with a size of (at least) Structure.size().
• Structure native memory can be read in bulk using Structure.getByteArray().
• Structures can be constructed from a pointer to native memory using the new Structure(Pointer p) constructor.
• The Structure.toArray() method creates an array of structures backed by a large, contiguous block of native memory.

I think your solution #2 is a rather efficient way of doing it, but your question indicates you'd like more type safety, or at least self-documenting code, in which case I'd point out a more "JNA way" of modifying #2 with two steps:

• Replace the new Memory(array.length * size) native allocation with the Structure.toArray() allocation from your solution #1.
  • You still have a length * size block of contiguous native memory and a pointer to it (array[0].getPointer()).
  • You additionally have pointers to the offsets, so you could replace mem.write(n * size, ... ) with array[n].getPointer().write(0, ... ).
• There is no getting around the memory copying, but having two well-commented lines which call getByteArray() and immediately write() that byte array seem clear enough to me.
  • You could even one-line it... write(0, getByteArray(0, size), 0, size), although one might argue if that's more or less clear.

So, adapting your method #2, I'd suggest:

// Make your collection an array as you do, but you could just keep it in the list 
// using `size()` and `list.get(n)` rather than `length` and `array[n]`.
MyStructure[] array = list.toArray(MyStructure[]::new);

// Allocate a contiguous block of memory of the needed size
// This actually writes the native memory for index 0, 
// so you can start the below iteration from 1
MyStructure[] structureArray = (MyStructure[]) array[0].toArray(array.length);

// Iterate the contiguous memory and copy over bytes from the array/list
int size = array[0].size();
for(int n = 1; n < array.length; ++n) {
    if(array[n] != null) {
        // sync local structure to native (using reflection on fields)
        array[n].write();
        // read bytes from the non-contiguous native memory
        byte[] bytes = array[n].getPointer().getByteArray(0, size);
        // write bytes into the contiguous native memory
        structureArray[n].getPointer().write(0, bytes, 0, bytes.length);
        // sync native to local (using reflection on fields)
        structureArray[n].read();
    }
}

From a "clean code" standpoint I think this rather effectively accomplishes your goal. The one "ugly" part of the above method is that JNA doesn't provide an easy way to copy fields between Structures without writing them to native memory in the process. Unfortunately that's the "JNA way" of "serializing" and "deserializing" objects, and it's not designed with any "magic" for your use case. Strings include built-in methods to convert to bytes, making such "magic" methods easier.

It is also possible to avoid writing the structure to native memory just to read it back again if you do the field-by-field copy as you implied in your Method #1. However, you could use JNA's field accessors to make it a lot easier to access the reflection under the hood. The field methods are protected so you'd have to extend Structure to do this -- which if you're doing that, the useMemory() approach is probably better! But you could then pull this iteration out of write():

for (StructField sf : fields().values()) {
    // do stuff with sf 
}

My initial thought would be to iterate over the non-contiguous Structure fields using the above loop, storing a Field.copy() in a HashMap with sf.name as the key. Then, perform that same iteration on the other (contiguous) Structure object's fields, reading from the HashMap and setting their values.

Answer 3

The solutions offered by Daniel Widdis will solve this 'problem' if one really needs to perform a byte-by-byte copy of a JNA structure.

However I have come round to the way of thinking expressed by some of the other posters - JNA structures are intended purely for marshalling to/from the native layer and should not really be used as 'data'. We should be defining domain POJOs and transforming those to JNA structures as required - a bit more work but deal with I guess.

EDIT: Here is the solution that I eventually implemented using a custom stream collector:

public class StructureCollector <T, R extends Structure> implements Collector<T, List<T>, R[]> {
    /**
     * Helper - Converts the given collection to a contiguous array referenced by the <b>first</b> element.
     * @param <T> Data type
     * @param <R> Resultant JNA structure type
     * @param data          Data
     * @param identity      Identity constructor
     * @param populate      Population function
     * @return <b>First</b> element of the array
     */
    public static <T, R extends Structure> R toArray(Collection<T> data, Supplier<R> identity, BiConsumer<T, R> populate) {
        final R[] array = data.stream().collect(new StructureCollector<>(identity, populate));

        if(array == null) {
            return null;
        }
        else {
            return array[0];
        }
    }

    private final Supplier<R> identity;
    private final BiConsumer<T, R> populate;
    private final Set<Characteristics> chars;

    /**
     * Constructor.
     * @param identity      Identity structure
     * @param populate      Population function
     * @param chars         Stream characteristics
     */
    public StructureCollector(Supplier<R> identity, BiConsumer<T, R> populate, Characteristics... chars) {
        this.identity = notNull(identity);
        this.populate = notNull(populate);
        this.chars = Set.copyOf(Arrays.asList(chars));
    }

    @Override
    public Supplier<List<T>> supplier() {
        return ArrayList::new;
    }

    @Override
    public BiConsumer<List<T>, T> accumulator() {
        return List::add;
    }

    @Override
    public BinaryOperator<List<T>> combiner() {
        return (left, right) -> {
            left.addAll(right);
            return left;
        };
    }

    @Override
    public Function<List<T>, R[]> finisher() {
        return this::finish;
    }

    @SuppressWarnings("unchecked")
    private R[] finish(List<T> list) {
        // Check for empty data
        if(list.isEmpty()) {
            return null;
        }

        // Allocate contiguous array
        final R[] array = (R[]) identity.get().toArray(list.size());

        // Populate array
        final Iterator<T> itr = list.iterator();
        for(final R element : array) {
            populate.accept(itr.next(), element);
        }
        assert !itr.hasNext();

        return array;
    }

    @Override
    public Set<Characteristics> characteristics() {
        return chars;
    }
}

This nicely wraps up the code that allocates and populates a contiguous array, example usage:

class SomeDomainObject {
    private void populate(SomeStructure struct) {
        ...
    }
}

class SomeStructure extends Structure {
    ...
}

Collection<SomeDomainObject> collection = ...

SomeStructure[] array = collection
    .stream()
    .collect(new StructureCollector<>(SomeStructure::new, SomeStructure::populate));

Hopefully this might help anyone that's doing something similar.