Is it necessary to consider the "BigEnd or LittleEnd" problem nowadays?

Question

We all were told there is big-end and little-end in computer system. But, almost my applications are running under x86 and arm architectures, which all are little-end designed system. In this way, is it necessary to consider this probelm always? It makes me confused when to take the endianness into account. Any tips or skillful suggesstion is appreciated~

Yup. You may find yourself interfacing with a program written in Java. — user4581301, Oct 07 '19 at 05:03
Yup. I shows up in the weirdest places. For instance the specification for ICC color profiles (color.org) which is a blob of bytes uses big endian in their format spec. which I had to accommodate. — doug, Oct 07 '19 at 05:07
Network byte order (used by TCP/IP, specified by RFC1700) is big endian. A bit hard to avoid that if your application does anything network related, or interfaces with applications written by others. — Peter, Oct 07 '19 at 05:08
ARM has both little- and big-endian mode, though I agree that in the mobile computing world little-endian is popular. — Ben Hirschberg, Oct 07 '19 at 05:08
@user4581301 That is an answer with a reason, i.e. a useful one. Why don't you make one? — Yunnosch, Oct 07 '19 at 05:27
@doug That is an answer with a reason, i.e. a useful one. Why don't you make one? — Yunnosch, Oct 07 '19 at 05:28
@Peter That is an answer with a reason, i.e. a useful one. Why don't you make one? — Yunnosch, Oct 07 '19 at 05:28
First of all, there is no problem. Handling BigEndian format conversion may be necessary indeed, but never by default. Even when you really need to handle it most likely it will be limited to a specific small portion of the program. Even when you write some Java interop or network stuff it makes no sense to make whole program BigEndian ready. Doing so will be overengineering. It is the same as other data representations, for example as fixed-point numbers. — user7860670, Oct 07 '19 at 05:39
@Peter Actually, some functions and interfaces in my program are local and encapsulated. I just wanna know, how you guys deal with these kind of annoying problems~ — Paul Wang, Oct 07 '19 at 05:40
There are some tricks to write code endian-agnostic, e.g. serializing an `uint32` with bitwise operations to split into bytes. [SO: How do I convert between big-endian and little-endian values in C++?](https://stackoverflow.com/a/10346064/7478597) Btw. recently I realized that floating point can have its individual endianess and it doesn't need to match that of integrals... ;-) — Scheff's Cat, Oct 07 '19 at 05:41
@P.Wang: What kind of problems? Endian issues are a problem with communicating with stuff *outside* of your program. — Nicol Bolas, Oct 07 '19 at 05:41
@NicolBolas Not only when _communicating with stuff outside of your program_. Short string optimization is a _internal program stuff_ and might require endian-aware implementation, such as [in libc++](https://github.com/llvm-mirror/libcxx/blob/master/include/string#L742). — Daniel Langr, Oct 07 '19 at 05:44
In the end it all comes down to the protocol being used for data interchange. If both sides agree on little endian, write it down in the protocol's definition document and everybody uses little endian. Period. If the protocol says all integers are signed, 32 bit and little endian, that's the protocol. It probably also says something like "All strings are proceeded with an integer specifying the length of the string." It's all an agreement. Write it down. Stick to it. But please try to avoid a monstrosity like [the SEISAN file protocol](http://seisan.info/). — user4581301, Oct 07 '19 at 05:51
@NicolBolas I mean, I'd like to find some skills to simplify my disigning. It's not a good experience to consider the endianness problem in each function of my program which deals with serilizing or writing file. Is there any more suggesstions or skillful tips? — Paul Wang, Oct 07 '19 at 05:53
@DanielLangr It's confused for me to take care of endianness problem everywhere. Nowadays, most of my program running environment is little-end, so~ I want to know you guys deal with this situation. Did u only consider little-end in specific environment. — Paul Wang, Oct 08 '19 at 04:02
@P.Wang Don't understand what do you mean by _everywhere_. A vast majority of C++ codes does not need to care about endianness. This is only a matter of memory layout, which applies to few problems only, such as serialization (network/file) or very specific in-memory optimizations (a particular form of short string optimization). Moreover, you can wrap these mechanisms in a library that will resolve endianness for you automatically (such as SSO implemented in libc++ or facebook/folly — users do not need to care). — Daniel Langr, Oct 08 '19 at 06:12
@DanielLangr well, I mean serialization in local file system. My application works on Windows. And I do believe most of my users' CPU are under X86 architecture. In this way, how do u think? — Paul Wang, Oct 08 '19 at 06:20
@P.Wang Then, it depends. There is no generic answer. If your program just stores some data to files and read them back on the same computer, I likely wouldn't care. However, if these files need to be transferred to a different system and opened there, then, I would care (basically you need to design a file format). You write _most of my users_, but _most_ does not mean _all_. This is very generic advice, in reality, it will be very different for a game app and a banking app. There are too many factors that play role here, and, in the end, you must decide. — Daniel Langr, Oct 08 '19 at 06:59
@P.Wang There is also another option that might be suitable for you. If you think that _all_ your users have LE, then suppose so, but trigger some error if a program is compiled for/run on a BE machine. — Daniel Langr, Oct 08 '19 at 07:08
@DanielLangr Words cannot express how thankful I am~ It's clear now — Paul Wang, Oct 09 '19 at 02:44

n. m. could be an AI · Accepted Answer · 2019-10-07T08:09:13.567

If you find yourself using any information about endianness of your architecture, you are either writing a compiler/standard library, or doing something wrong.

Binary data formats do have endianness which applications need to take into account. Many existing data formats use big endian representation.

So an application in general only deals with endianness at interfacce boundaries. This is ideally encapsulated in platform APIs like htonl and ntohl, but sometimes these are unavailable/unsuitable and an implementation needs to provide its own. It still can be completely oblivious to platform endianness. For example instead of

 uint32_t var;
 file.read(&var, sizeof(var));
 #if LITTLE_ENDIAN
     swap_byte_order(var);
 #endif

one can write

uint32_t var; byte buf[4];
file.read(buf, sizeof(buf));
var = buf[3] | buf[2]<<8 | buf[1]<<16 | buf[0]<<24;

A modern compiler comiles the last line to a single bswap instruction on an x86. It works regardless of the endianness of the architecture, so no preprocessor madness and no configuration headaches.

For more info, read The byte order fallacy.

score 2 · Answer 2 · answered Oct 07 '19 at 05:41

2

Yes; network byte order disagrees with x86, among other things.

Never read bytes to/from serialization as a memcpy as a general rule. Always repack and arrange the bytes structurally or mathematically. This is annoying, but fixes both endianness and many other problems down the pipe.

Good compilers, if you are actually writing them like memcpy and the compiler can prove it, will optimize.

answered Oct 07 '19 at 05:41

Yakk - Adam Nevraumont

262,606
27
330
524

Thanks for your answer. It's helpful to me. Btw, is there any skill to simplify this annoying problem? such as, interface desiging, or other useful tips. – Paul Wang Oct 07 '19 at 05:56
Not to mention PowerPC and sharp edges like [Is vec_sld endian sensitive?](https://stackoverflow.com/q/46341923/608639). Linux runs on PPC in little-endian mode, while AIX runs in big-endian mode. Source code written for them must do their own endian gyrations even with high level code. – jww Oct 07 '19 at 05:57

Is it necessary to consider the "BigEnd or LittleEnd" problem nowadays?

2 Answers2