4

Suppose I have a C-style array of type unsigned char: 

unsigned char * c = (unsigned char *) malloc(5000 * sizeof(unsigned char));
for(int i = 0; i < 5000; i++) 
    c[i] = (unsigned char) ((i >> (i%4 * 8)) & 0xFF);

Suppose I have a pointer offset to a position which starts a 4 byte integer: 

// pseudo code
unsigned int i = c + 10; // 10 = pointer offset, let's say.

If I want to load i up with the correct number, I can do: 

unsigned int i = (*(c+10) << 24) + (*(c+11) << 16) + (*(c+12) << 8) + (*(c+13));

But shouldn't I just be able to, somehow, do this using casts?

// pseudo code -- I haven't gotten this to work yet: 

int i = (unsigned int) (*((void *)(c+10));

// or maybe
int i = *((unsigned int*)((void *)(c+10)));

In short, what is the cleanest, most effective way to transition the four bytes to an unsigned int in a C-style byte array?

Chris
  • 28,822
  • 27
  • 83
  • 158

3 Answers3

7

The proper way to do this is to use memcpy: 

unsigned int i;
std::memcpy(&i, c + offset, sizeof(unsigned int));

On architectures that support unaligned variable access (like x86-64), this will be optimized into a simple pointer dereference, but on systems that don't support unaligned access (such as ARM), it will do the proper thing to get the value out.

See for example: https://gcc.godbolt.org/z/l5Px4G . Switch the compiler between gcc for x86 and arm and see the difference in instructions.

Keep in mind the idea of endianness if you're getting the data from some external source. You may have to flip the bytes of the integer around for the value to make sense.

Paul Belanger
  • 2,354
  • 14
  • 23
  • 1
    Why would you use platform-specific code where an equally efficient platform-independent option exists? – David Schwartz Sep 28 '18 at 19:21
  • 3
    What about my approach is platform-specific? – Paul Belanger Sep 28 '18 at 19:22
  • 1
    The value of `i` depends on the paltform's endianness. – David Schwartz Sep 28 '18 at 19:24
  • 1
    @DavidSchwartz we've been through that in your answer. The question if it depends on it or not can not be answered without knowing how the buffer was initially populated. If it was created by the same application, it does not. – SergeyA Sep 28 '18 at 19:25
  • Hence the comment on endianness at the bottom. You can do a compile-time check for the platform endinaness and do a `htonl` if needed. – Paul Belanger Sep 28 '18 at 19:26
  • @SergeyA We only need to know that the buffer was not initially populated with platform-specific code. We can, and should, avoid platform-specific code when we can and there's no evidence that any platform-specific code is required here. You're hearing hoof falls and reasoning about zebras. – David Schwartz Sep 28 '18 at 19:29
  • @DavidSchwartz I have no idea what you are talking about. Although the question clearly lacks information on how this buffer came to existence, one of the most common ways would be through entity-serialization of a struct - which is a perfectly reasonable thing to do within application. There is also 0 (zero) platform specific code in the example shown, so I am completely puzzled by your comments, – SergeyA Sep 28 '18 at 19:32
  • @PaulBelanger If the source data is native endian, use `memcpy`, if source is little (or middle) endian, then use shifts, if the source is big endian then use shifts or `hton` family of functions. That way you don't need multiple versions for different endianness. – eerorika Sep 28 '18 at 19:47
  • The question specifically asked about approaching the problem without using shifts. – Paul Belanger Sep 28 '18 at 19:49
  • 1
    @PaulBelanger The asker is hindered by the [XY-problem](http://xyproblem.info/). It's OK to tell them that their desire to not use shifts is silly. – eerorika Sep 28 '18 at 19:51
  • 1
    Want to add: this works, unless I am using a char * array with netbyte order. Is there a work around using memcpy that can reverse the ordering of the bytes? Is there a way to manually for loop through the four bytes such that simd is triggered? I am not sure there is a guarantee that an expression with bit shifts will have the same amount of optimization as a for loop... – Chris Sep 28 '18 at 19:52
  • 1
    You can use the [`ntohl`](https://linux.die.net/man/3/htonl) function for this. Just pass the integer through it after the memcpy. On x86-64, the ntohl compiles to a single `bswap` instruction: https://gcc.godbolt.org/z/PZrJSp – Paul Belanger Sep 28 '18 at 19:54
  • FYI: Since ARMv6 (an architecture from 2002), ARM CPUs support unaligned access. – geza Sep 28 '18 at 20:24
  • Looks like that's correct! The point still stands though that the memcpy call will generate correct code for all CPUs: https://gcc.godbolt.org/z/bIQ6Rw – Paul Belanger Sep 30 '18 at 13:08
3

No, you shouldn't. Adding an offset that's not a multiple of an object's size to a pointer to an allocated object can result in a pointer that the platform cannot dereference. It's simply not a pointer to an unsigned int.

On some platforms, performance will be atrocious. On some platforms, the code will fault.

In any event, the shifts and adds are very clear and easy to understand. The cast is more confusing and requires understanding the platform's byte ordering. So you're not making things better, simpler, or clearer.

David Schwartz
  • 179,497
  • 17
  • 214
  • 278
  • 1
    This is an opinion. What is an answer? – Chris Sep 28 '18 at 19:14
  • Disagree. It all depends on how this char buffer was populated. And on THE platform (x86_64) the performance will be exactly the same as with aligned data. – SergeyA Sep 28 '18 at 19:16
  • 1
    As far as standard c++ is concerned, this answer is correct. Any concern that it *might* work on certain platforms is betting on undefined behavior or platform/compiler specific guaranties. Edit : This comment is referring to the first paragraph. The last paragraph is an opinion. – François Andrieux Sep 28 '18 at 19:16
  • @bordeo "*Adding an offset that's not a multiple of an object's size to a pointer to an allocated object can result in a pointer that the platform cannot dereference.*" is an opinion? – David Schwartz Sep 28 '18 at 19:18
  • @SergeyA I don't follow you. What depends on how the buffer was populated? – David Schwartz Sep 28 '18 at 19:18
  • The byte ordering thing. If the buffer was populated by memcopying the int, it is the shifts that require you to understand byte ordering, while the cast is very straightforward way of converting back. – SergeyA Sep 28 '18 at 19:21
  • @SergeyA No. If the buffer was populated by memcopying the int, you would have to understand the platform's byte ordering to understand what was in the buffer. You would even have to understand the platform's byte ordering to make a sensible unit test. That would be needlessly platform-specific code and the OP seems to know how not to make that mistake. – David Schwartz Sep 28 '18 at 19:27
  • @DavidSchwartz what??? I have no idea what you are talking about. If the int was memcopied into buffer, best way to extract it is to memcopy it back. Period. – SergeyA Sep 28 '18 at 19:29
  • @SergeyA Why are you assuming there's some platform-specific code elsewhere? Isn't the more reasonable assumption, if we're going to make one, that we're talking about platform-independent code? Sure, it's remotely possible this is a platform-specific issue, but there's no evidence of that in the question and platform-independent answers should be the norm, maybe with a note of any platform-specific issues that might arise. – David Schwartz Sep 28 '18 at 19:30
  • @DavidSchwartz I am not sure what you are referring to as 'platform specific code'. Since in other answer you seem to imply that memcpy platform-specific, I suggest we continue our conversation there. – SergeyA Sep 28 '18 at 19:33
  • 2
    Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/180960/discussion-between-david-schwartz-and-sergeya). – David Schwartz Sep 28 '18 at 19:43
3

But shouldn't I just be able to, somehow, do this using casts?

No, there's no cast that's guaranteed to work.

Note that there are many representations for an integer. How to convert an array of bytes to an integer object depends on how the integer is represented in the array. If an integer is converted to an array of bytes and sent over a network for example, you cannot know whether the receiving computer uses the same representation.

One consideration is how negative numbers are represented. Luckily 2's complement is such a ubiquitous representation that we can usually ignore this. In your case though, it's even less important since you're converting an unsigned integer.

A more relevant consideration is byte endianness.

If you know that the array is in the same representation as is used by the CPU that executes the program, then you could copy the bytes using std::memcpy:

unsigned int i;
static_assert(sizeof i == 4);
std::memcpy(&i, c + 10, sizeof i);

This works correctly regardless of the endianness used by the CPU, as long as the source data is in the same representation.

Your suggestion (*(c+10) << 24) + ... is correct (or appears to be, I didn't thoroughly check) if the representation of the byte array is big endian. The suggestion is wrong if the array is little or some other endianness.

This approach is useful when receiving data over the network, since it does not rely on the representation being same as the executing CPU.

eerorika
  • 232,697
  • 12
  • 197
  • 326
  • Ah, And I can memcpy right into the int pointer? (assuming I have declared the int) – Chris Sep 28 '18 at 19:24
  • Why do you care about sign if the question clearly has unsigned ints? – SergeyA Sep 28 '18 at 19:27
  • @SergeyA because I assume the question to be a simplified version of a more general question. – eerorika Sep 28 '18 at 19:28
  • What is the `assert(sizeof i == 4);` for? – NathanOliver Sep 28 '18 at 19:43
  • @NathanOliver The wording of the question, specially the per-byte bitshifting implementation, implies a 4 byte `int`. It may not be necessary for this solution in the general case, but OP's case certainly seems to depend on it. If this solution was compiled on a different platform with larger or smaller bytes, it would fail. – François Andrieux Sep 28 '18 at 19:45
  • @NathanOliver Well, OP stated that the array contains a 4 byte integer, while `int` is not guaranteed to be 4 bytes. On the other hand, `memcpy` relies on the endianness matching the CPU, so it would not be far fetched to assume the size to match native as well. – eerorika Sep 28 '18 at 19:46
  • @user2079303 If the solution requires `sizeof(int)==4` then how about using `int32_t` instead of `int`? – François Andrieux Sep 28 '18 at 19:47
  • OK. That makes sense. I forgot the OP said the `int` in the array was 4 bytes – NathanOliver Sep 28 '18 at 19:47
  • @FrançoisAndrieux well, `CHAR_BIT` is not guaranteed to be 8, so `sizeof(int32_t)` is not guaranteed to be 4 :) – eerorika Sep 28 '18 at 19:49
  • @user2079303 That's true, but then `int32_t` is unlikely to be defined. But lets say `CHAR_BIT` is 16 and there *is* a `int32_t`, then `int32_t` will read 2 bytes of 16 bits. Assuming this is a cross-platform data transmission situation (otherwise, the discussion doesn't matter) then it would be fine. Or, no worse than the current `int` solution. The point is, there is exactly 32 bit of data to read. Edit : Actually, your solution would fail in that situation since you would read `CHAR_BIT * 4` bits, which isn't what's expected. – François Andrieux Sep 28 '18 at 19:52
  • I may be wrong if the data comes *from* copying an `int`. The correct number of bits may actually be 32. I guess it's unclear what the intention is in case of an non 8 bit `char` system. – François Andrieux Sep 28 '18 at 19:56
  • @FrançoisAndrieux I'm just going by the specification in the question which states that in consists of 4 bytes. If that is an error and the real specification is 4 octets, then `int32_t` would probably be ideal indeed. Note that there's no reason a system with `CHAR_BIT` of 16 wouldn't support `int32_t` other than simply supporting only an older version of the standard. – eerorika Sep 28 '18 at 19:57