Best practice to build C++ program with optimization while keeping debugability in production env

Question

I am writing a C++ server program that will be deployed to *nix systems (Linux/macOS). The program sometimes runs into segfault in production environment, and I would like to collect some information like core dump file. But I have no idea what the best practice is for doing this:

1. I would like to make the program perform best.
2. I would like to analyze the core dump in production env offline if it really happens.

I learn that there are some things I could try:

1. There is RelWithDebInfo and Release for CMAKE_BUILD_TYPE, but it seems they have different optimization level. So I assume RelWithDebInfo build performs not as well as Release build. (“RelWithDebInfo uses -O2, but Release uses -O3“ according to here What are CMAKE_BUILD_TYPE: Debug, Release, RelWithDebInfo and MinSizeRel?)
2. Tools like objectcopy/strip allow you to strip debug information from a binary (How to generate gcc debug symbol outside the build target?)
3. Printing stack trace when handling SIGSEGV signal (How to automatically generate a stacktrace when my program crashes)

I am new to deploy a production C++ server program, and I would like to know the answers for the following questions:

1. What is the recommended build type to use in this case, RelWithDebInfo or Release?
2. Compared to choosing different build type, when do I need to use tools like strip?
3. If I create a Release build binary for production deployment, when the Release build generates a core dump in production environment, can I later use the using the same revision of source code to build a RelWithDebInfo binary and use (gdb + RelWithDebInfo binary + Release build core dump) for core dump analysis?
4. Is it common to turn on core dump in production env? If it is not a good practice, what could be the recommended approach for collecting information for troubleshooting, printing the stack trace when crashing?

In general, I would like to know how C++ programs is recommended to be build for production, allowing it to be best optimized while I am still able to troubleshoot it. Thanks so much.

Answer 1

This will be a rather general answer.

1. If you are still having reliability issues then go with RelWithDebInfo. Alternatively, you can override -O2 optimization to get compiler to optimize all the way.
2. You need to strip when debug info is present. This doesn't change any of the stuff that actually gets executed, but removes things that make debugging much easier. You can still debug stripped executables, but it is more difficult to understand what is going on.
3. No, due to different optimization levels. If the only difference between the two was that one was stripped and the other not, then yes. But with different optimization levels the resulting assembly will actually be different.
4. Enabling core dumps in production environments is usually advised against, mostly for security reasons. For example, core dump may contain plain-text passwords, session tokens, etc. These are all things that others should not see. If you have total control of the machine where this is running then this concern is somewhat smaller. Another concern is disk space usage. Core dump can be huge, depending on what your program is doing. If you have fixed core file name then at least there will enver be more than one file, but if you have a name that includes timestamp and/or PID, then you can have multiple files, each of them taking lots (meaning n GB) of space. This can again lead to problems.

The general rule is (or should be) that you consider release environment as hostile. Sometimes that is true, sometimes it is not -- here generality can't apply because only you know your specific situation.

I always deploy my stuff fully optimized. I will only include debug info if a program is particularly problematic, because it makes it easy to either run it with or attach to it using gdb.

The downside of full optimization is that things sometimes look a bit different than the code you wrote. Order of things may change, some things may not happen at all, and you may observe that some functions don't really exist as proper standalone functions because compiler decides they are better of being inlined. These are changes I observed, but there are probably others as well.

Answer 2

Recently I learn that there are tools like Google Breakpad which will generate crash report with minidump format that could be collected and analyzed in production environment. I don't give it a try yet but it could be useful for this exact purpose.