How can I change the byte orders of a struct?

Question

How can I change the order of a packed structure in C or C++?

struct myStruct {
  uint32_t A;
  uint16_t B1;
  uint16_t B2;
} __attribute__((packed));

The address 0x0 of the structure (or the LSB) is A.

My app communicates with hardware and the structure in hardware is defined like this:

struct packed {
  logic [31:0] A;
  logic [15:0] B1;
  logic [15:0] B2;
} myStruct;

But in SystemVerilog the "address 0x0" or more accurately the LSB of the structure is the LSB of B2 = B2[0].

The order is reversed.

To stay consistent and to avoid changing the hardware part, I'd like to inverse the "endianness" of the whole C/C++ structure.

I could just inverse all the fields:

struct myStruct {
  uint16_t B2;
  uint16_t B1;
  uint32_t A;
} __attribute__((packed));

but it's error-prone and not so convenient.

For datatype, both SystemVerilog and Intel CPUs are little-endian, that's not an issue.

How can I do it?

I didn't get it. How is this gonna reverse the C struct? The packed attribute is already doing what I need. — Alexis, Oct 19 '20 at 10:43
If the machine does not do what you want, that is store the bits in the correct order you need to take care of this yourself. Had you had a look at the `ntoh()` and `hton()` functions? — alk, Oct 19 '20 at 10:46
Isn't there a way to generate the C struct from the SystemVerilog definition automatically? — Ted Lyngmo, Oct 19 '20 at 10:46
Just break it down to bytes and shift them to the correct positions. There's hundreds of examples about that here on the site. — Lundin, Oct 19 '20 at 10:49
@alk It's in the right order, just the way we define structure in both languages, the LSB is not placed at the same place. — Alexis, Oct 19 '20 at 10:49
@TedLyngmo That's not what I want. I'd like to get the same order of fields in both structures. — Alexis, Oct 19 '20 at 10:50
@AlexF It's just an example, it can be uint8_t or whatever, not necessarily multiple of 32b. I still don't get how using bitfield instead of the struct packed is a solution to my question. I don't wanna packed the struct by myself with bitfields. — Alexis, Oct 19 '20 at 10:58
@Alexis You may not be able to have the same order of the fields in both structures if the languages do not agree where the LSB is. — Ted Lyngmo, Oct 19 '20 at 10:59
@alk I perfectly know those functions, I don't understand how it can be a solution to my question. I don't have any problem with endianness of each field. — Alexis, Oct 19 '20 at 11:01
@TedLyngmo finally a comment that makes sense. Alright, that's a language definition, no hope for an attribute. A macro that inverse the order of each field should work. Thanks! — Alexis, Oct 19 '20 at 11:04
Yeah, I guess a macro could do but it seems odd that the SystemVerilog fields are reversed. It goes against common practice I'd say. — Ted Lyngmo, Oct 19 '20 at 11:08
In hardware it makes more sense to have the LSB at the right/end because in the end it's not a memory but just an array of bits. Also the convention of a byte is to read the 8bits word with the LSB at the right. — Alexis, Oct 19 '20 at 11:14
System verilog operates 4-state values. As such its bit representation has nothing to do with 'c' representation. Also there is no order of fields in respect to the real memory defined in verilog standard. You need to use verilog api conversion functions to map data structs. They depends on the method you use to communicate. — Serge, Oct 19 '20 at 11:14
@Serge I'm curious about that verilog api that exchanges the fields order of a struct. — Alexis, Oct 19 '20 at 11:45
@Alexis there are 2 main standard api types for verilog: VPI and DPI (in System Verilog only). Both allow data exchange and invoke functions cross boundary. You should start with DPI which is the most modern one. It is not a stand-alone tool but is a part of a standard verilog toolkit (from any vendor). Of course, you can also use IO files to do data exchange using verilog file features. Ascii will work ok, but byte-oriented will have the same issues as in your question. — Serge, Oct 19 '20 at 13:17
@Serge ok now I got it, sorry if I wasn't explicit enough. My question is related to actual hardware (SoC w/ DMA w/ embedded linux), not verification. — Alexis, Oct 19 '20 at 13:49
You haven't explained your problem in enough detail. SystemVerilog is for describing hardware and the endianness of a structure has nothing to do with its layout--unless you are using the DPI to cross the language boundary and need to pass data. — dave_59, Oct 19 '20 at 14:57
@Alexis it has nothing to do with verilog then. I suggest to remove the 'System Verilog' tag then. Also reference to verilog and its data structure is very confusing. You should re-formulate your question as related to 'c' only. — Serge, Oct 19 '20 at 17:59
@Alexis I had this exact same issue and followed the same thought process as you. The only solution I came up with was to reverse the ordering of the fields in the systemverilog declaration. I wish there was an option in SV to declare that members of a packed struct should be in reverse order, it would make stuff like this much easier. — John M, May 06 '23 at 15:03

eerorika · Accepted Answer · 2020-10-19T13:52:47.793

3

How can I change the byte orders of a struct?

You cannot change the order of bytes within members. And you cannot change the memory order of the members in relation to other members to be different from the order of their declaration.

But, you can change the declaration order of members which is what determines their memory order. The first member is always in lowest memory position, second is after that and so on.

If correct order of members can be known based on the verilog source, then ideally the C struct definition should be generated with meta-programming to ensure matching order.

it's error-prone

Relying on particular memory order is error-prone indeed.

It is possible to rely only on the known memory order of the source data (presumably an array of bytes) without relying on the memory order of the members at all:

unsigned char* data = read_hardware();
myStruct s;
s.B2 = data[0] << 0u
     | data[1] << 8u;
s.B1 = data[2] << 0u
     | data[3] << 8u;
s.A  = data[4] << 0u
     | data[5] << 8u
     | data[6] << 16u
     | data[7] << 24u;

This relies neither memory layout of the members, nor on the endianness of CPU. It relies only on order of the source data (assumed to be little endian in this case).

If possible, this function should also ideally be generated based on the verilog source.

edited Oct 19 '20 at 13:52

answered Oct 19 '20 at 11:13

eerorika

232,697
12
197
326

`Relying on particular memory order is error-prone indeed.` is this serious? When you do hardware with memory mapped, the memory order is the only thing you can rely on. – Alexis Oct 19 '20 at 11:52
I'm not a big fan of over-complicated things. How this can be easier than just reverting the order of the fields in the struct? I don't see any added-value at all. – Alexis Oct 19 '20 at 11:56
@Alexis `is this serious?` Yes. `When you do hardware with memory mapped, the memory order is the only thing you can rely on.` That doesn't mean that you would have to rely on memory order of structs, because that can be avoided as demonstrated. It is best to rely on as minimal set of assumptions as possible. `How this can be easier than just reverting the order of the fields in the struct?` Is the reversed order guaranteed by some specification? This code works the same regardless of the order of the members and regardless of CPU endianness. – eerorika Oct 19 '20 at 12:05
Why not relying on the packed struct type in C/C++? `you would have to rely on memory order of structs` But that's exactly what I want to do. I dare to believe the gcc/C/C++ specs define the fields order of a packed struct and that doesn't change depending on the moon. `best to rely on as minimal set of assumptions as possible` I'm for using the power of the tools at my disposal. – Alexis Oct 19 '20 at 12:45
`regardless of the order of the members and regardless of CPU endianness` I agree since the code redo what the compiler should do for you. I'm looking for an easier way, not a more complicated one. Being able to map a pointer of struct directly to a piece of memory is priceless and compile-time whereas your solution is runtime and I believe can be quite expensive depending on the data size. I don't have the portable requirement. Thanks for your solution anyway, I'll keep that pattern in mind. – Alexis Oct 19 '20 at 12:45
Extra: is this a kernel dev pattern? I'm curious to know for which field/problematic this pattern is used and how often. I've never seen it. – Alexis Oct 19 '20 at 12:49
1

`Why not relying on the packed struct type in C/C++?` There is no such thing as "packed" struct type in either language standard. As such, standard conformance is one reason. Another reason is portability (byte-endianness). If you had sub-byte divisions, then that would be one more reason in general case (because allocation of bitfields is implementation defined). – eerorika Oct 19 '20 at 13:09
1

And the *worst* part of packed structures is that [they can very well cause bugs and crashes](https://stackoverflow.com/a/46790815/918959). – Antti Haapala -- Слава Україні Oct 19 '20 at 13:17
1

`Being able to map a pointer of struct directly to a piece of memory is` something that cannot work if there is difference in the layout of the struct and the layout of the memory. Either because source data lacks alignment, or because the struct has padding, or order of members is wrong, or because byte endianness differs. In case the layout doesn't match, the shown solution should be equally fast as any other way to re-order the bytes. – eerorika Oct 19 '20 at 13:30
In the lucky case that the layout matches, if you can give up the support for padded structures and can assume sufficient alignment of the buffer, then you can copy the converted structure back onto the buffer, which can then be optimised into no-op. Example: https://godbolt.org/z/bTn974 – eerorika Oct 19 '20 at 13:30
@Alexis `which field/problematic this pattern is used` This is fairly common when ever structures are de/-serialised for file storage or streamed buffers such as network communication. – eerorika Oct 19 '20 at 13:32
@AnttiHaapala That's a whole different story. And good doc: https://www.keil.com/support/man/docs/armclang_intro/armclang_intro_xxq1474359912082.htm – Alexis Oct 19 '20 at 13:50
Thanks for the godbolt, that's interesting but only for gcc from 8.1. I keep it in my bookmark! – Alexis Oct 19 '20 at 14:08
@Alexis yes. *"If a member of a structure or union is packed and therefore does not have its natural alignment, then to access this member, you must use the structure or union that contains this member. You must not take the address of such a packed member to use as a pointer, because the pointer might be unaligned. Dereferencing such a pointer can be unsafe even when unaligned accesses are supported by the target, because certain instructions always require word-aligned addresses."* – Antti Haapala -- Слава Україні Oct 19 '20 at 14:10

John Bollinger · Answer 2 · 2020-10-19T11:41:36.893

How can I change the order of a packed structure in C or C++?

C specifies that the members of a struct are laid out in memory in the order in which they are declared, with the address of the first-declared, when converted to the appropriate pointer type, being equal to the address of the overall struct. At least for struct types expressible in C, such as yours, conforming C++ implementations will follow the same member-order rule. Those implementations that support packed structure layout as an extension are pretty consistent in what they mean by that: packed structure layouts will have no padding between members, and the overall size is the sum of the sizes of the members. And no other effects.

I am not aware of any implementation that provides an extention allowing members to be ordered differently than declaration order, and who would bother to implement that? The order of members is well-defined. If you want a different order, then the solution is to change the declaration order of the members.

If VeriLog indeed orders the members differently (to which I cannot speak) then I think you're just going to need to make peace with that. Implement it as you need to do or as otherwise makes the most sense, document on both sides, and move on. I'm inclined to think that the number of people who ever notice that the declaration order differs in the two languages will be very small. As long as appropriate the documentation is present, those that do notice won't be inclined to think there's an error.

`who would bother to implement that` me for example. I discard all the other irrelevant judgemental comments. Whatever the number of people who notice it, my question is legit and it could have been an implemented automatic solution. One would think the bitfields feature makes even less sense, but it exists. seriously why would you define `1b` for an `unsigned int` field?! (rhetoric) Thanks for your time anyway. — Alexis, Oct 19 '20 at 12:58
I am not being judgemental, @Alexis, and I think you are missing the point. The question is not whether *you* might *use* such a feature, but rather whether a *compiler writer* would *implement* it. I assert that no, a typical compiler writer would not be likely to implement an extension to achieve something that the language already has a standard, simple mechanism for doing. And none has done, so far as I know. The broader point, which responds directly to your question, is that such a feature would indeed be an extension. Neither C nor C++ provides a built-in mechanism for what you ask. — John Bollinger, Oct 19 '20 at 14:31

score 0 · Answer 3 · answered Oct 20 '20 at 05:17

You know I just looked AMD does in it's open source drivers to handle endianness.

First of all they detect if the system is big endian/little endian using cmake.

#if !defined (__GFX10_GB_REG_H__)
#define __GFX10_GB_REG_H__

/*
*    gfx10_gb_reg.h
*
*    Register Spec Release:  1.0
*
*/

//
// Make sure the necessary endian defines are there.
//
#if defined(LITTLEENDIAN_CPU)
#elif defined(BIGENDIAN_CPU)
#else
#error "BIGENDIAN_CPU or LITTLEENDIAN_CPU must be defined"
#endif

union GB_ADDR_CONFIG
{
    struct
    {
#if defined(LITTLEENDIAN_CPU)
        unsigned int                       NUM_PIPES : 3;
        unsigned int            PIPE_INTERLEAVE_SIZE : 3;
        unsigned int            MAX_COMPRESSED_FRAGS : 2;
        unsigned int                       NUM_PKRS  : 3;
        unsigned int                                 : 21;
#elif defined(BIGENDIAN_CPU)
        unsigned int                                 : 21;
        unsigned int                       NUM_PKRS  : 3;
        unsigned int            MAX_COMPRESSED_FRAGS : 2;
        unsigned int            PIPE_INTERLEAVE_SIZE : 3;
        unsigned int                       NUM_PIPES : 3;
#endif
    } bitfields, bits;
    unsigned int    u32All;
    int             i32All;
    float           f32All;
};

#endif

Yes there is some code duplication as mentioned above. But I'm not aware of a universally better solution either.

No relation with my question, read the comments. Even though it's a struct, it's actually a 32b word (bitfields). Moreover I don't understand their code since endianness is only based on 8b and they also inverse the order of internal fields `MAX_COMPRESSED_FRAGS, PIPE_INTERLEAVE_SIZE, NUM_PIPES` which the order should be endianness independent. Unless it's a bitfield characteristic. — Alexis, Oct 20 '20 at 22:36

score 0 · Answer 4 · answered Aug 04 '22 at 19:37

Independent of the endian issue, I wouldn't recommend C++ bit fields for this kind of purpose, or any purpose in which you actually need explicit control of bit alignment. A long time ago, the decision to put performance over portability ruined this possibility. Alignment of bit fields (and structs in general for that matter) is not well defined in C++, making bit fields useless for many purposes. IMO would be better to let programmers make such decisions for optimization, or implement a strictly portable (non-machine dependent) packed keyword. If this means the compiler has to emit code that combines multiple shift-and operations once in a while, so be it.

As far as I know, the only general solution for this kind of thing is to add a layer that implements bit fields explicitly using shift-and logic. Of course this will likely ruin performance because you really want the conditionals to be handled at compile time, which is ironic because performance is what motivated this situation in the first place.

What do you mean by bit field? a simple packed structure like in my question or literally using bit field `:N`? — Alexis, Aug 04 '22 at 22:23
Well, it's a little off topic I suppose, since the OP was just about byte order. A bit field is an integer field with a specific number of bits, perhaps 3 or 19 bits for example. A "correct" implementation would give the programmer full predictable control over the allocation of bits. C prioritizes optimization over control, so the allocation is undefined. — Ken Seehart, Aug 24 '22 at 17:21

How can I change the byte orders of a struct?

4 Answers4