19

I have recently started using CMake, and was trying to build a GUI application, that doesn't have the console window on Windows. So in my CMakeLists.txt file, I did this:

if("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU" OR "${CMAKE_CXX_COMPILER_ID}" STREQUAL "Clang")
    add_executable(${EXECUTABLE_NAME} main.cpp)
elseif("${CMAKE_CXX_COMPILER_ID}" STREQUAL "MSVC")
    add_executable(${EXECUTABLE_NAME} WIN32 main.cpp) #WIN32 So the console window does not open on Windows
endif("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU" OR "${CMAKE_CXX_COMPILER_ID}" STREQUAL "Clang")

With this, the solution worked, and the console window does not open on Windows. However, this comes at a cost. When I try to build the solution, I realize that I have to change the signature of the function to WinMain, so I changed my main code to the following:

#ifdef _WIN32
#include <Windows.h>
int WINAPI WinMain(HINSTANCE, HINSTANCE, PSTR, int) //Windows signature since creating WIN32 application without console
#else
int main()
#endif
{
    // ... GUI code
}

Unfortunately, I absolutely detest this, since it ruins the whole point of using CMake. I don't want to have to change anything in my code that is based on different platforms. This leads me to my question. How do I set the C++ application entry point to main() on Windows when making a GUI application without having to set it manually in Visual Studio? Can I do this directly in CMake using a cross-platform method? Or will I have to use the #if/#else/#endif solution? The only improvement to the solution above is using a macro MAIN_FUNCTION that does the preprocessor conditional. I want to avoid this as well.

On the other hand, is there another way to get rid of the console window in a GUI application on Windows that I didn't know of using CMake without using the WIN32 option?

Arnav Borborah
  • 11,357
  • 8
  • 43
  • 88
  • you need set entry point not to `main` but to `mainCRTStartup` (if you use ansi) or to `wmainCRTStartup` (if you use unicode) – RbMm Aug 17 '17 at 00:25
  • `main()` is the standard entry point for cross-platform console apps, and `WinMain()` is the standard entry point for GUI apps on Windows. Vendor runtime libraries are usually compiled to have separate `.obj` files that depend on this separation. – Remy Lebeau Aug 17 '17 at 00:38
  • 1
    You can set properties on the target dependent on the compiler used, e.g. `if(MSVC) set_target_properties(${EXECUTABLE_NAME} PROPERTIES WIN32_EXECUTABLE TRUE) endif()` – vre Aug 17 '17 at 06:18
  • 2
    Correction to bounty message above. A solution which does _NOT_ require creating a macro. – Arnav Borborah Mar 12 '18 at 19:53

2 Answers2

22

The solution is to add set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} /SUBSYSTEM:WINDOWS /ENTRY:mainCRTStartup") before add_executable

It hides the console while still allowing you to have the usual int main() as the entry point.

Gabriel
  • 829
  • 5
  • 12
  • 1
    I'm glad it worked! I use it for my project's release build. Personally, I like having the console there for debugging. – Gabriel Mar 17 '18 at 14:44
  • A better option today is to use cmake features for this. Example `target_link_options(target PRIVATE "/SUBSYSTEM:WINDOWS")`. https://cmake.org/cmake/help/latest/command/target_link_options.html – John Hanley May 28 '23 at 06:24
5

You're confusing two things here, but they are closely related.

The console that appears is a result of an application which has a Win32 header IMAGE_OPTIONAL_HEADER::Subsystem value of WINDOWS_CUI instead of WINDOWS_GUI. This is a Win32 thing, and it applies to all executables regardless of the language they're written in.

The entry point signature is a compiler-specific choice. It's the entry function called by the language runtime, not the OS. The OS calls the entry function of the language runtime, which first initialized that runtime and then hands off control to your entry point.

Now the VC++ compiler uses the CRT as the runtime. And that CRT runtime indeed uses two different signatures for your entry point. Obviously the implementation of std::cin has to work with WINDOWS_CUI, that's sort of the point of a Command-Line User Interface. But the same CRT also works with WINDOWS_GUI.

Here's where things get complex. You can actually change the Subsystem of a compiled application from CUI to GUI. The CRT won't mind, it's compatible with both subsystems. But since it's done to a compiled application, the call from one part of the application (CRT startup) to another (your entry point) isn't affected. This is a Win32 change, not a C++ change.

To get back to CMake: this subsystem change can be done either after CMake, or as a custom post-build step.

MSalters
  • 173,980
  • 10
  • 155
  • 350
  • *The entry point signature is a compiler-specific choice* - under windows no. the signature of *exe* EP is common - `ULONG __stdcall function(void* Peb)` which function will be used for entry point is set by [`/ENTRY:function`](https://docs.microsoft.com/en-us/cpp/build/reference/entry-entry-point-symbol) linker option. if we explicit not use this option, linker by default use `[w]mainCRTStartup` (cui) or `[w]WinMainCRTStartup` (gui). compiler here unrelated at all. and note that `[w]WinMain` (called from `[w]WinMainCRTStartup`) or `[w]main` (called from `[w]mainCRTStartup`) not entry points. – RbMm Mar 14 '18 at 01:58
  • @RbMm: Read the second paragraph again. I make it clear that I'm talking about the entry point into the application, called by the language runtime. – MSalters Mar 14 '18 at 07:38
  • @MSalters Could you clarify _what_ this subsystem change might look like? – Arnav Borborah Mar 15 '18 at 11:13
  • @ArnavBorborah - at first subsystem change - are windows attach console to process. at second, if you not use explicit `/ENTRY:function` - linker select entry point based on default - [`/ENTRY (Entry-Point Symbol)`](https://learn.microsoft.com/en-us/cpp/build/reference/entry-entry-point-symbol) – RbMm Mar 16 '18 at 09:43
  • @RbMm Ok, I get what you are doing there. So should I use [`set_target_properties`](https://cmake.org/cmake/help/v3.1/command/set_target_properties.html) to set `/ENTRY`? – Arnav Borborah Mar 16 '18 at 10:39
  • @ArnavBorborah - all what you need - explicit set `/ENTRY:mainCRTStartup`. as result this and will be entry point of your *exe*. and this entry point call your `main`. so you will able have `main` only (without #if/def and WinMain). this not depend from subsystem actually. but how set `/ENTRY:mainCRTStartup` in cmake - i dont know – RbMm Mar 16 '18 at 10:49
  • @ArnavBorborah - so i think - you need change not subsystem (this affect your executable - say add console window if you select CUI subsystem), but all what need - explicit change entry point to `/ENTRY:mainCRTStartup` as i comment at very begin in 2017 – RbMm Mar 16 '18 at 10:52
  • @ArnavBorborah you need use `LINK_FLAGS` property to set `/ENTRY:mainCRTStartup` – RbMm Mar 16 '18 at 10:55
  • @ArnavBorborah: My idea is incompatible with RbMm's suggestions. I literally mean "change the field in the PE header of the compiled executable". I.e. read the EXE is as a data file, change that byte, write it back. – MSalters Mar 16 '18 at 11:05
  • @MSalters If I changed the compiled executable, then how would that affect the signature of `main`? Once the executable is compiled (Changed with the new byte), how would I be able to edit the signature? I know RbMm's suggestion currently works, so I am inclined towards using that right now. – Arnav Borborah Mar 16 '18 at 11:16
  • @ArnavBorborah: The point is that you compile it with the desired signature. – MSalters Mar 17 '18 at 09:45
  • 1
    @MSalters Oh, I see. You're saying to use `main` as a signature, and then change the byte that would otherwise cause an error after compilation. That makes sense! However, the solution provided Alexander is lot easier to implement for me as your solution would require parsing the PE header and finding the byte to change it. Thanks anyway for the in-depth explanation of the process! – Arnav Borborah Mar 17 '18 at 13:03