JNA How to populate a pointer-to-structure field within a structure to be PASSED to the native library?

Question

I need to pass a JNA structure to the native layer that contains a pointer-to-structure field (may contain zero-or-more structures).

Here is the 'parent' structure:

public class VkRenderPassCreateInfo extends Structure {
    public int attachmentCount;
    public VkAttachmentDescription.ByReference pAttachments;
}

(Other fields, @FieldOrder and ByReference/Value classes omitted for brevity)

And here is the 'child' structure:

public class VkAttachmentDescription extends Structure {
    public int flags;
    // ... lots and lots of other simple fields
}

As per the JNA documentation (here) the pointer-to-array field should be a Structure.ByReference field.

From other posts the standard approach to populating this field is:

1. init the field to a structure-by-reference

2. allocate an array of structures from the field using Structure::toArray

3. populate the array

So:

// Init structure fields
renderPass.pAttachments = new VkRenderPassCreateInfo.ByReference();
renderPass.attachmentCount = size;

// Allocate memory
VkAttachmentDescription[] attachments = (VkAttachmentDescription[]) renderPass.pAttachments.toArray(size);

// Populate array
for(int n = 0; n < size; ++n) {
    attachments[n].flags = ...
    // and so on for other fields
}

1 - Is this the correct approach for initialising and allocating a pointer-to-structure field within a structure? Seems a lot of mucking around?

2 - The above works fine for fiddling sized structures but some of the ones I'm dealing with have a huge number of fields, sub-structures, etc. I had assumed I could just build an array of JNA structures on the Java side and set those directly into the parent structure, but the toArray approach means I have to then copy everything to the generated array? Is there a better/simpler approach that means I don't have to create and copy data that I essentially already have on the Java side?

3 - JNA provides a StringArray helper class that handles a similar case for an array of strings within a structure:

// Array of strings maintained on the Java side
List<String> strings = ...

// Nice and easy means of populating the JNA structure
structure.pStrings = new StringArray(strings.toArray(String[]::new));
...

// Send the data
library.fireandForget(structure);

This is sort-of what I'm trying to achieve with the structure code above but obviously is only for the string case - are there other similar helpers that I've missed?

Please note that the above is passing the structures to the native layer, I am not trying to retrieve anything.

EDIT 1: Just to qualify the point of this question - although the above works it results in a buge amount of boiler-plate code for all but the most trivial cases. I'm struggling to work out the simplest/best way of building a complex graph of structures to be passed to a native method. There seems a shortage of examples or tutorials, or maybe I'm just not asking the right question (?) Any pointers to examples, tutorials or example code of passing structures containing pointers to other structures would be very appreciated.

EDIT 2: So I've tried numerous approaches all of which result in Illegal memory access errors when I invoke the native library.

The data that I want to send is constructed by the application - it could be a builder pattern, selections by the user, etc. In any case the result is a list of VkAttachmentDescription that I then need to send as a pointer-to-structure field in the 'parent' VkRenderPassCreateInfo.

The reason for using the JNA VkAttachmentStructure on the Java side is that some these structures contain a large number of fields. i.e. calling Structure::toArray and then populating the resultant array field-by-field is just not tenable: the amount of code would be huge, error-prone and brittle to change (e.g. forget to copy a new field). I could create another class to abstract the JNA class but that would just move the problem.

Here is what the code is doing:

// Application builds the attachments
final List<VkAttachmentDescription> attachments = ...

...

// At some point we then send the render pass including the attachments

// Populate the render pass descriptor
final VkRenderPassCreateInfo info = new VkRenderPassCreateInfo();
info.pAttachments = ??? <--- what?
// ... other fields

// Send the descriptor
library.sendRenderPass(info);

Attempt 1: Naively set the pointer-to-structure to an array:

final VkRenderPassCreateInfo info = new VkRenderPassCreateInfo();
final var array = attachments.toArray(VkAttachmentDescription.ByReference[]::new);
info.pAttachments = array[0];
library.sendRenderPass(info);

Result is a memory access error, I didn't expect this to work!

Attempt 2: Use Structure::toArray(int) and set the field to the first element

final VkAttachmentDescription.ByReference[] array = (VkAttachmentDescription.ByReference[]) new VkAttachmentDescription.ByReference().toArray(attachments.size());

for(int n = 0; n < size; ++n) {
    array[n] = attachments.get(n);
}

info.pAttachments = array[0];

library.sendRenderPass(info);

Same result.

Attempt 3: Use Structure::toArray(array)

There is an alternative toArray method in Structure that takes an array but it doesn't seem to do anything different to calling the integer version?

Attempt 4: Copy field-by-field

final VkAttachmentDescription.ByReference[] array = (VkAttachmentDescription.ByReference[]) new VkAttachmentDescription.ByReference().toArray(attachments.size());

for(int n = 0; n < size; ++n) {
    array[n].field = attachments.get(n).field;
    // ...lots of other fields
}

info.pAttachments = array[0];

library.sendRenderPass(info);

This works but is nasty.

I'm obviously completely missing something about JNA. My main sticking point is that Structure::toArray creates an array of empty structures that have to be populated field-by-field, but I already have the array of structures with everything populated - how can I set the pointer-to-structure field to that array (i.e. an equivalent of the StringArray helper)? It seems such a simple thing to do in my head but I simply cannot find any examples of how to do what I want (other than trivial ones that copy field-by-field).

The other thing that is bothering me is that fact that the parent structure field has to be ByReference which means every other structure in the code has to be by-reference? Again it just feels like I'm doing this all wrong.

Answer 1

The issue you need to solve (and the source of the Illegal memory access errors) is that the C-side code accepting your array is expecting a Pointer to a contiguous block of memory. In C, you only need the memory address of the first element, plus the size offset; to access array[1] you find the memory of array[0] and offset by the size of the structure.

In your case, you have allocated non-contiguous memory for each structure in this block:

// Application builds the attachments
final List<VkAttachmentDescription> attachments = ...

Each VkAttachmentDescription is mapped to its own memory, and attempting to read the memory at the end of the first structure causes the error. If you do not have control over which memory is used when these VkAttachmentDescription objects are instantiated, you will end up duplicating your memory requirements, and having to copy over the native memory from the non-contiguous block to the contiguous block.

Edited to add: As pointed out in your other answer, if you've only worked with the VkAttachmentDescription Structure on the Java side and not passed it to a C function, the native memory may not have been written. The below solutions based on Pointer.get*() methods read directly from the C memory, so they would require a write() call to be made at some point.

Assuming you have no choice other than to start with a List<VkAttachmentDescription>, the first thing you need to do is allocate the contiguous memory that C needs. Let's get the byte sizes we need:

int size = attachments.size();
int bytes = attachments.get(0).size();

We need to allocate size * bytes of memory.

You have two options here: directly allocate the memory using a Memory object (a subclass of Pointer) or use Structure.toArray. For the direct allocation:

Memory mem = new Memory(size * bytes);

We can directly use mem as a Pointer if we define the reference like this:

public class VkRenderPassCreateInfo extends Structure {
    public int attachmentCount;
    public Pointer pAttachments;
}

Then it's a simple:

info.pAttachments = mem;

Now all that is left is to copy the bytes from the non-contiguous memory into your allocated memory. We can do it byte by byte (easier to see what's happening at the byte level on the C side):

for (int n = 0; n < size; ++n) {
    Pointer p = attachments.get(n).getPointer();
    for (int b = 0; b < bytes; ++b) {
        mem.setByte(n * bytes + b, p.getByte(b));
    }
}

Or we can do it structure by structure:

for (int n = 0; n < size; ++n) {
    byte[] attachment = attachments.get(n).getPointer().getByteArray(0, bytes);
    mem.write(n * bytes, attachment, 0, bytes);
}

(Performance tradeoff: array instantiation overhead vs. Java<-->C calls.)

Now that the buffer is written, you can then send it to C, where it expects the array of structures, and it won't know the difference... bytes are bytes!

Edited to add: I think it's possible to change the native memory backing using useMemory() and then directly write to the new (contiguous) location. This code is untested but I suspect might actually work:

for (int n = 0; n < size; ++n) {
    attachments.get(n).useMemory(mem, n * bytes);
    attachments.get(n).write();
}

Personally, since we're just making a copy of something that already exists, I would prefer this Memory-based mapping. However... some coders are masochists.

If you want to be a bit more "type safe" you can use the ByReference class declaration inside the structure and create the Structure array using toArray(). You've listed in your code one way of creating the array using the ByReference type. That works, or you can also create it with the (default ByValue) type and then extract the Pointer to the first element later to create the ByReference type when assigning it to the structure field:

VkAttachmentDescription[] array = 
    (VkAttachmentDescription[]) new VkAttachmentDescription().toArray(attachments.size());

Then you can set it like this:

info.pAttachments = new VkAttachmentDescription.ByReference(array[0].getPointer());

In this case, it's a bit more complex copying the values over from the List (of structures backed by individually allocated memory blocks) to the array (of contiguous memory) because the memory mapping is more narrowly typed, but it follows the same pattern as for the Memory mapping. One way, which you've discovered, is to manually copy over every element of the Structure! (Ugh.) Another way which might save you from some copy/paste errors is to use Reflection (what JNA does under the hood). That's a lot of work, too, and duplicates what JNA does, so that's ugly and error-prone. However, it is still possible copy over the raw native bytes from the non-contiguous memory block to the contiguous one. (In which case... why not just go straight to Memory but my bias is showing.) You can iterate over bytes like in the Memory example, like this:

for (int n = 0; n < size; ++n) {
    Pointer p = attachments.get(n).getPointer();
    Pointer q = array[n].getPointer();
    for (int b = 0; b < bytes; ++b) {
        q.setByte(b, p.getByte(b));
    }
}

Or you can read the bytes in chunks like this:

for (int n = 0; n < size; ++n) {
    byte[] attachment = attachments.get(n).getPointer().getByteArray(0, bytes);
    array[n].getPointer().write(0, attachment, 0, bytes);
}

Note that I haven't tested this code; it writes to the native side but not the Java structure so I think it will work as is, but you may need an array[n].read() call at the end of the above loop to read from C to Java in case there's a built-in Java to C copy that I'm not aware of.

In response to your "parent structure field has to be ByReference": As shown above, a Pointer mapping works and allows a bit more flexibility at the cost of "type safety" and maybe (or not) "readability". You don't need to use ByReference elsewhere as I've shown with the toArray() where you only need it for the Structure field (which you could just define as a Pointer and completely eliminate the need for ByReference ... but if you're doing that why not just copy to the Memory buffer? I'm beating a dead horse here!).

Finally, the ideal solution if you know how many elements you will eventually have (or an upper bound on that number), would be to instantiate the array using contiguous memory at the very beginning. Then instead of creating a new instance of VkAttachmentDescription you could just fetch a pre-existing one from the array. It's okay if you over-allocate and don't use them all as long as you use them contiguously from the beginning. All you're passing to C is the # of structures and the address of the first one, it doesn't care if you've got extra bytes.

Answer 2

Here is a static helper to performs the structure copying approach outlined above:

    /**
     * Allocates a contiguous memory block for the given JNA structure array.
     * @param structures Structures array
     * @return Contiguous memory block or <tt>null</tt> for an empty list
     * @param <T> Structure type
     */
    public static <T extends Structure> Memory allocate(T[] structures) {
        // Check for empty case
        if(structures.length == 0) {
            return null;
        }

        // Allocate contiguous memory block
        final int size = structures[0].size();
        final Memory mem = new Memory(structures.length * size);

        // Copy structures
        for(int n = 0; n < structures.length; ++n) {
            structures[n].write(); // TODO - what is this actually doing? following line returns zeros unless write() is invoked
            final byte[] bytes = structures[n].getPointer().getByteArray(0, size);
            mem.write(n * size, bytes, 0, bytes.length);
        }

        return mem;
    }

The helper can be used to populate pointer-to-structure fields, e.g:

info.pAttachments = StructureHelper.allocate(attachments.toArray(VkAttachmentDescription[]::new));
info.attachmentCount = attachments.size();

This appears to work but I'm worried about the write that appears to be needed in the copying loop. Without this the byte[] extracted from the structure is zeroes. What is write actually doing? The doc says copying to native memory but I can't make head-nor-tail of what the actual code is doing.

Should I be releasing this memory afterwards?

Is there any alternative method to get the structure memory?