Accurately calculate the compiler added padding in a struct

Question

I currently have the following structure

struct foo {
    u32 bar;
    u8 baz[1];
}

The sizeof(struct foo) is turning out to be 8 due to padding.

If I want the size of the struct that's accurate (in this case 5), how would I go about calculating in a "scale-able" way (i.e. not doing something like sizeof(foo.bar) + sizeof (foo.baz))?

Sounds nonsensical. What's supposed to happen with padding that's added *between* members? — Hans Passant, Jan 15 '14 at 22:59
What's the reason for wanting the non padded size? If you wanted your struct to not be padded you can always use compiler hints to do that `__attribute__((packed))` but the non padded size is useless in most cases — Jesus Ramos, Jan 15 '14 at 22:59
I'd likely use a `#pragma` that set the struct alignment to 1, thus eliminating any padding. Failing that, IIRC, the machine-word size will be what the padded struct will be a multiple of. I.e 32bit = 4byte alignment. 64bit = 8byte alignment. An array of 4byte aligned structs will be faster to access than a smaller, 1byte aligned one. — enhzflep, Jan 15 '14 at 23:03
@HotLicks: "scale-able" = generic. Shouldn't really depend knowing the name of elements of the struct. Basically I want to be able to put this into a macro that applies for a few more structs. — R.D., Jan 15 '14 at 23:03
Perhaps if you explain why you need this info we could be of more assistance. — Hot Licks, Jan 15 '14 at 23:06
@JesusRamos Basically this struct is the layout of a packet. The other client receiving the packet expects sizeof(foo) to be 5. Instead of manually calculating this (which would be icky because there are ton of structs like which, which are parts of various other packets), I was looking for a better way to do i.t — R.D., Jan 15 '14 at 23:06
Why would the client packet receive a struct of size 5 if the compiler is padding to 8? You have to enforce the same layout when sending binary data on both ends or you encode it in a different way. I fail to see how that size is going to help. — Jesus Ramos, Jan 15 '14 at 23:07
R.D. serialize the members to the stream instead of bitwise copy into a struct. Alternatively, use `#pragma` to set the alignment for that struct so that the compiler doesn't add padding. — Mooing Duck, Jan 15 '14 at 23:08
Why not just make your "packet" structs "packed" all the time? You would need to use a little care accessing them, but the same structure size would be understood by all. — Hot Licks, Jan 15 '14 at 23:08
Don't use structs for layouts of packets. Write proper serialize and deserialize routines. — R.. GitHub STOP HELPING ICE, Jan 15 '14 at 23:09
How will your "packets" be described? Some sort of descriptive structure -- eg, a list of type/size pairs? — Hot Licks, Jan 15 '14 at 23:11
@HotLicks They really vary by the type. There could be another packet that'd look like `struct foo { u32 bar1; u8 baz1[1]; u32 bar2; u8 baz2[1];}`, in which case I'd want to calculate the size as 10 rather than 16. Anyway for now, I'm going with the packed struct route, wish it didn't involved compiler specific crud :/. — R.D., Jan 15 '14 at 23:14
So how do you describe that, so that both ends are working from the same description? — Hot Licks, Jan 15 '14 at 23:17
Don't use structs as network protocols. You are introducing at least six separate dependencies. It's only reliable if you deploy the same object code at both peers. Define a wire protocol in octets and write yourself a library to send and receive it. — user207421, Jan 15 '14 at 23:20
@HotLicks As it turns out, the other side is using a different compiler that has decided to not to pad it. I'm working on an embedded platform with multiple CPUs that could implement different ISAs. I think the "client" is running on an ARM core in Thumb mode — R.D., Jan 15 '14 at 23:20
So you say the other side uses the same C-language .h file you do, but they always have "packed" turned on? — Hot Licks, Jan 15 '14 at 23:23
If you say you want the size of elements without listing them all (in case they are a lot) you can do sizeof(int) + sizeof(double) + ... multiply each one for the number of elements of that type — fernando.reyes, Jan 15 '14 at 23:25
The fact that they are using a different compiler bears out my comment. Don't do this. You are introducing dependencies on (1) the compiler (2) the compiler version (3) the CPU architecture (4) the compiler options in use (5) the surrounding #pragmas and (6) the `struct` definition itself. — user207421, Jan 15 '14 at 23:38
You could use variadic templates to construct the struct's members, and calculate the sum of all the individual types' sizes and store it as a static member of the struct. This would be portable so long as you could guarantee the size of the input data is in a specific order, which it sounds like you can't. :/ — Colin Basnett, Jan 15 '14 at 23:38
@R.D. Instead of reinventing what user `R.` suggested, you could use some code generation tool (e.g. s.th. like [google protobuf](http://code.google.com/p/protobuf/)) that is able to solve all the various de-/serialization pitfalls and problems for you. As your question stands, it sounds like an XY-problem for me. — πάντα ῥεῖ, Jan 15 '14 at 23:41
You still haven't explained how you expect both sides to understand the structure. If you have to write unique code on each end that renders your question moot. If you have a way to describe the data to both ends, one could work from that description rather than the structs. — Hot Licks, Jan 16 '14 at 00:20

score 3 · Accepted Answer · answered Jan 15 '14 at 23:44

There is no way in standard C to calculate the total of the sizes of the members of a struct without listing them in the source code (which might be done with some assistance from preprocessor features or by using other code [compiled and executed at compile-time of the final code] to generate the final code). This is because standard C does not provide any way to iterate through the members of a struct or otherwise inspect its composition without using the names of its members.

I would make the same claim for C++ except that some horrible hack might be possible with templates and whatnot. I lean toward that not being possible, but I have not examined all the new features in the latest C++ standard yet.

In any case, calculating the total size would be insufficient for the stated purpose of serializing and deserializing the members (that is, converting the members to bytes in a network package and vice-versa) because you would still need to individually convert the members, not just know the total of their sizes.

Options for approaching the goal include:

Use implementation-specific features to pack the struct so that it contains no padding. Then you can write and read the bytes of the struct to perform the serialization and deserialization. You will still need to ensure that the types match between the sending and receiving systems (same width integers, same format floating-point values, same character set encoding, et cetera).
Write code for run-time handling of desired types. It would take a description of a struct in some format such as a list of types and would have cases for each of those types. Each case would contain compile-time code for the type (e.g., sizeof(float)), but the code would dispatch between the cases at run-time. So you would need to prepare a description of the struct in the format used by this code.

I think I have implemented some relevant C++11 solution to persist packed data in flash memory. The SO questions and answers that lead to the productive code can be found [here](http://stackoverflow.com/questions/18242003/creating-an-api-metaprogramming-dsl-for-static-initialization-of-a-layout-desc) and [here](http://stackoverflow.com/questions/18251815/creating-an-array-initializer-from-a-tuple-or-variadic-template-parameters) — πάντα ῥεῖ, Jan 15 '14 at 23:54
One needs to have some handwritten (or even easily generatable) mapping code to deal with what I mentioned of course ... — πάντα ῥεῖ, Jan 16 '14 at 00:02

2785528 · Answer 2 · 2014-01-16T00:59:23.497

"...in a scalable way?" -- no can do, C does not have memory layout semantics.

Note: I have never found a compiler to give an incorrect value for "sizeof(T)",

The compiler is telling you that an array of N struct foo objects, as you have presented here, will take up 8*N bytes.

I have, however, on one project, coded to force the 'proper', expected, and shall we say most compact 'packing' of an object, without using a pragma. (I am of the opinion that pragmas are non-portable. However, every compiler that I have investigated which contained pragmas, seemed to provide excellent results. The pragma was occasionally not spelled the same)

In other words, if you really need a 5 byte foo object it is possible, and I think relatively easy, to accomplish. But the following scheme quickly gets tiresome if your object foo grows to have a lot of fields.

In my embedded software project, the remote processor used a different processor, and different compiler, on different build machine, and they used their compiler's pragma pack.

Our software had to describe an object that complied with their packing, because thousands of that subsystem had already shipped. Our code had to be able to interact. These systems communicated with binary packets sent across a proprietary back plane. (Neither C nor C++ provide memory layout symantics. Ada did, but does anyone care anymore?)

We tried a number of solutions, each with pro's and con's.

The following is styled after 1 of maybe 5 successful mechanisms ... and this one is easy to remember.

Two steps:

Declare your object using array of byte.
Add the appropriate setters and getters, 1 of each per field.

Note: you will learn endian-ness, look into the macro's ntoh and hton

namespace fooNameSpace 
{
   struct foo
   {

   private:
      uint8_t pfd[5]; //packed_foo_data

   public:
      void barSet(uint32_t value)
         {
            pfd[0] = static_cast<uint8_t>((value & 0xff000000) >> 24);  // msbyte
            pfd[1] = static_cast<uint8_t>((value & 0x00ff0000) >> 16);
            pfd[2] = static_cast<uint8_t>((value & 0x0000ff00) >> 8);
            pfd[3] = static_cast<uint8_t>((value & 0x000000ff) >> 0);  // lsbyte
         }; // you will need to fix for endian of target

      uint32_t barGet(void)
         {
            uint32_t value = 0;
            value |= pfd[0] << 24; // msbyte
            value |= pfd[1] << 16;
            value |= pfd[2] << 8;
            value |= pfd[3] << 0; // lsbyte
            return (value);
         }; // fix for endian of target

      void     bazSet(uint8_t value) { pfd[4] = value; }      
      uint8_t  bazGet(void)          { return pfd[4]; };
   };
}

Some of my team tried to create a union ... and found out it got confusing when the union must match the remote compiler, target, endian-ness on the local host and compiler. Kinda confused the emulator, too.

Accurately calculate the compiler added padding in a struct

2 Answers2