Calculate member offset of unknown type

Question

I want to get the offset of a struct's member. I know this has been asked multiple times and the answer is always the mighty offsetof. Well, my case is a little different: I need the offset of an unknown type. That is for example:

void fill_struct(void* unknown)
{
    ...
}

The only thing I will know from unknown is the order in which types are set. i.e.

int
int
float
...
string

And the main problem here is align/padding, since I don't know a way to calculate it nor if there is a way at all.

This kind of question is often replied with: why would you want to do that?

For those people: I'm implementing a JSON parser in C++, and faced a problem (representing multiple type arrays), and my solution is to map the array's values into a custom struct.

I accept feedback regarding to that solution but I'm mainly interested in my question being answered

You can start with a map of variants instead of guessing the layout. — Captain Obvlious, Aug 28 '15 at 01:38
You cannot calculate offsets from a void pointer in C, since it is a type of unknown size, as you say. Functions that accept a void pointer type must also take the size of the type it points to. `void*` is more usually used for a return argument. See http://stackoverflow.com/questions/692564/concept-of-void-pointer-in-c-programming — Weather Vane, Aug 28 '15 at 01:43
Do not add C tag for C++ questions! This one might present a C++ specific solution. Although until now it is not clear what you want to accomplish. `void *` does not point to a structured object and both languages are statically typed, so you have to know the current type anyway. — too honest for this site, Aug 28 '15 at 01:44
@WeatherVane I understand that, but I'm able to cast it to char* and use pointer arithmetic. <- that works in some cases and others are flawed because of align/padding — Rodolfo, Aug 28 '15 at 01:47
@Olaf Its C++, I'm just using void* for receiving any kind of pointers as parameters knowing its structure (but again, the problem is the align/padding) — Rodolfo, Aug 28 '15 at 01:49
@Olaf, I agree it's a hack... but nice stuff comes from hack... besides, this has academic purposes mostly. — Rodolfo, Aug 28 '15 at 01:51
Alignment is implementation specific. And some compilers have extensions to selectively disable padding per struct. So I don't see how it's possible to work out the field offsets based on just the info you have (field type and order). Unless you apply further constraints and assumptions on the system. — kaylum, Aug 28 '15 at 02:03
"but nice stuff comes from hack" as does the biggest crap and many of the security holes we are facing everywhere. I'd say the negative effects are the vast majority. And, sorry, but I'm not up to "academic" nonsense (i.e. without any gain in knowledge or practical use). Good luck with further "research". — too honest for this site, Aug 28 '15 at 02:08
@AlanAu: I do not understand the problem. If he has the field-types and order, he also has a proper struct definition he can cast to. If that is serialized data, there is already a message layout description he can deserialize into a proper struct (hmm, .. XY-problem?). — too honest for this site, Aug 28 '15 at 02:10
@Olaf I could not have access to the struct definitions since they would be custom (user defined), as this is intended to be a library — Rodolfo, Aug 28 '15 at 02:12
Rodolfo: so a few things come to mind. why are you trying to use a struct to represent array types? why wouldn't you use either a C union type, or a C++ variant type like boost::variant to represent "general" json types? and use a std::vector to represent a json array? besides this, why don't you use any of the many wondeful existing C++ json reading/writing libraries? i can personally vouch for this one from the makers of boost::spirit https://github.com/cierelabs/json_spirit — Chris Beck, Aug 28 '15 at 02:14
@Olaf I might have misunderstood OP requirements but my reading is that OP is trying to serialise an unknown struct type. The field types and ordering is known. But the alignment is not known. So two structs can have exactly the same fields in the same order but packing may have been enabled on one of the structs and not the other. In which case the offsets will be different despite their same field types and ordering. — kaylum, Aug 28 '15 at 02:14
@AlanAu: But that would require a completely different approach (that's why is more and more smell an XY-problem). To serialize JSON, one uses a generic struct per JSON-element which can basically hold the few different JSON types and links to the next. In C++ one can use classes or a struct with union and type-tag similar to C. I completely agree with your concerns in general, but I do not see how this is really related to parsing JSON. You just build your object -tree while parsin, each node one JSON-value/subtree. How could you provide dynamic types in C++ otherwise? — too honest for this site, Aug 28 '15 at 02:17
@ChrisBeck: I have never seen boost::variant, will keep that in mind. I cannot use std::vector for JSON arrays since they can have multiple types. And why am I not using a third-party library? because I want to implement a technique that claims to be the best http://www.infoq.com/articles/HIgh-Performance-Parsers-in-Java-V2 (but in c++) — Rodolfo, Aug 28 '15 at 02:20
Rodolfo: boost::variant is the most natural way in C++ to represent what are sometimes called "algebraic union types", i.e. a data that is of exactly one of the types in the union. and it does this without causing memory or allocation problems, it is a fully-stack allocated object. you can use a std::vector of boost::variant for json arrays, even though they are heterogenous, and then you won't have to do any alignment or padding calculations. in modern C++ you generally never have to do that, it's a high-level programming language. even in C you usually don't have to do that. — Chris Beck, Aug 28 '15 at 02:23
Use templates instead of ugly `void*` functions and then you can just use `offsetof` — Red Alert, Aug 28 '15 at 02:23
@Olaf: It might be an XY problem, but I was curious about align/padding calculation as priority. I know there are plenty of solutions to my problem, I just wanted an intuitive way to represent JSON within C++ — Rodolfo, Aug 28 '15 at 02:24
@ChrisBeck: I suspect OP wants to build true C++ structs dynamically at run-time from the JSON. That would work in Python, but not in C++. — too honest for this site, Aug 28 '15 at 02:25
I mean he could just use std::map or similar for each node of the json. That's basically what the python solution is also? Python doesn't have true structs at all — Chris Beck, Aug 28 '15 at 02:26
@Olaf: I think it can be done with meta-programming, and it would not be runtime — Rodolfo, Aug 28 '15 at 02:27
@RodolfoCastilloMateluna: No offence, but you seem to have quite some missconceptions. Perhaps you should read about the difference between [statically and dynamically](https://en.wikipedia.org/wiki/Statically_typed#Type_checking) typed languages. — too honest for this site, Aug 28 '15 at 02:28
@ChrisBeck: Python is completely object-oriented, much further than C++ or Java. you can easily have a simple struct-like container class. That would be run-time expandable/modifyable (Python is strong dynamically typed). — too honest for this site, Aug 28 '15 at 02:30
Olaf: but it will not be allocated contiguously like a C / C++ struct, it will be heap allocated similar to a std::map or std::unordered_map. and lookup will be much harder than adding a fixed offset to a pointer, you'll have to traverse the data structure at runtime. — Chris Beck, Aug 28 '15 at 02:31
@RodolfoCastilloMateluna: Is the JSON parsed at compiler-time or run-time? — too honest for this site, Aug 28 '15 at 02:31
@Olaf: Not offended, I do know what are statically/dynamically typed languages... I insist, if I generated code from a user's struct definition, so I can get its offset with **offsetof**, it would be at compile-time, and the types would be well-defined — Rodolfo, Aug 28 '15 at 02:32
@ChrisBeck: No. Python's most important type is likely maps/dicts. IIRC, the JSON module uses also dicts to store the entries - of course: `name: value`. That's why I do not get what OP actually wants. I did not say it is fast, flexibility always comes at the cost of overhead. — too honest for this site, Aug 28 '15 at 02:34
Rodolfo: if thats how you want to do it, then maybe you should look into boost::fusion? boost::fusion is a very powerful library that allows you to do things like, iterate over the sequence of members of a struct, which you can't do normally in the language. it is based on template meta programming. it's one of the easier of those kinds of libraries to use imo — Chris Beck, Aug 28 '15 at 02:34
@Olaf: the JSON would be parsed at run-time, but I expect the user to know the way its arranged, so at compile-time I would already have user-defined array layouts — Rodolfo, Aug 28 '15 at 02:34
@ChrisBeck: I will take a look at that (I was never atracted to boost black magic, guess I have to give up some day) — Rodolfo, Aug 28 '15 at 02:36
@RodolfoCastilloMateluna: But if you have the layouts already, you also have the structs. The JSON as a pure text-serialization has no specific binary layout anyway. Ok, I'm off, need some sleep. Have fun! — too honest for this site, Aug 28 '15 at 02:37
Yes. As I suspected: XY-problem. Perhaps you take the information you got and think over it first. The pointers from @ChrisBeck sound promising. — too honest for this site, Aug 28 '15 at 02:45
Rodolfo: I've always been more attracted to the boost black magic than the `void *` `offsetof` black magic. You pick your poison I suppose :D — Chris Beck, Aug 28 '15 at 02:47
@ChrisBeck: Suppose I was always attracted to good ol' hacks — Rodolfo, Aug 28 '15 at 02:51

Calculate member offset of unknown type

0 Answers0