Confusing flags passed to MSVC through NVCC with CMake

Question

I have a CMake file which I'm using to build some CUDA on Windows (NVCC/MSVC). I'm trying to set the MSVC warning level to /W4, using:

add_compile_options("$<$<COMPILE_LANGUAGE:CUDA>:--compiler-options=/W4>")

Building with CMake 3.9, I get this warning:

(CudaBuildCore target) ->
  cl : Command line warning D9025: overriding '/W4' with '/W3'

In CMake 3.15, the policy was changed to not automatically set /W3 in the CUDA flags, but with that version I get:

(CudaBuildCore target) ->
  cl : Command line warning D9025: overriding '/W4' with '/W1'

If I do the build step with --verbose I see the following with 3.9:

  "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin\nvcc.exe" -gencode=arch=compute_30,code=\"compute_30,compute_30\" --use-local-env -ccbin "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64" -x cu  -IC:\Users\eddi\Documents\temp\thur\sw\include -IC:\Users\eddi\Documents\temp\thur\sw\shared\common\include -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\include"     --keep-dir x64\Debug -maxrregcount=0  --machine 64 --compile -cudart static --compiler-options=/W4 --compiler-options= -Xcompiler="/EHsc -Zi -Ob0" -g   -D_WINDOWS -D"FISH_BUILD_TYPE=\"DEBUG\"" -D"CMAKE_INTDIR=\"Debug\"" -D"CMAKE_INTDIR=\"Debug\"" -D_MBCS -Xcompiler "/EHsc /W3 /nologo /Od /Fdfish.dir\Debug\fish.pdb /FS /Zi /RTC1 /MDd /GR" -o fish.dir\Debug\fish_cuda.obj "C:\Users\eddi\Documents\temp\thur\sw\fish_cuda.cpp"

and with 3.15:

  "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin\nvcc.exe" -gencode=arch=compute_30,code=\"compute_30,compute_30\" --use-local-env -ccbin "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64" -x cu  -IC:\Users\eddi\Documents\temp\thur\sw\include -IC:\Users\eddi\Documents\temp\thur\sw\shared\common\include -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\include"     --keep-dir x64\Debug -maxrregcount=0  --machine 64 --compile -cudart static --compiler-options=/W4 --compiler-options= -Xcompiler="/EHsc -Zi -Ob0" -g   -D_WINDOWS -D"FISH_BUILD_TYPE=\"DEBUG\"" -D"CMAKE_INTDIR=\"Debug\"" -D"CMAKE_INTDIR=\"Debug\"" -D_MBCS -Xcompiler "/EHsc /W1 /nologo /Od /Fdfish.dir\Debug\fish.pdb /FS /Zi /RTC1 /MDd /GR" -o fish.dir\Debug\fish_cuda.obj "C:\Users\eddi\Documents\temp\thur\sw\fish_cuda.cpp"

Spoiler: these are identical, except for the -Xcompiler "/EHsc /W3 /nologo /Od /Fdfish.dir\Debug\fish.pdb /FS /Zi /RTC1 /MDd /GR" portion.

Try as I might, I cannot find where this set of flags is being introduced (by CMake et al.), and hence can't start working out how to change the behaviour.

EDIT1: Adding color to the story... I grep'd my CMake install for /nologo and found several .cmake files with lines like:

   35:   set(CMAKE_CL_NOLOGO "/nologo")

I changed them all to variations of

   35:   set(CMAKE_CL_NOLOGO "/nologo_EDD")

But none of them turn up in the confusing set of options. So either I've missed something or they're being introduced from something outside CMake?

EDIT2: @squareskittles said:

you can remove the default flag from the CMAKE_CXX_FLAGS_INIT variable

Fair enough, I can see the logic to that: it's simple and it directly addresses my immediate issue. But it seems a bit blunt and doesn't address where the flags are coming from in the first place.

Anyway, objections aside, I added this ugly morsel to my script:

message( "--------------------------- ${CMAKE_CXX_FLAGS_INIT}")
string (REGEX REPLACE "/W[0-4]" "" CMAKE_CXX_FLAGS_INIT "${CMAKE_CXX_FLAGS_INIT}")
message( "--------------------------- ${CMAKE_CXX_FLAGS_INIT}")

With CMake 3.9, I see:

---------------------------   /DWIN32 /D_WINDOWS /W3 /GR /EHsc
---------------------------   /DWIN32 /D_WINDOWS  /GR /EHsc

as hoped. But with 3.15, I see:

---------------------------   /DWIN32 /D_WINDOWS /GR /EHsc
---------------------------   /DWIN32 /D_WINDOWS /GR /EHsc

ie, the /W3 flag isn't there in the first place (let alone a /W1 flag) due to the policy change. But even worse, the primary issue still persists in both cases:

nvcc.exe ... --compiler-options=/W4 -Xcompiler="/EHsc -Zi -Ob0" ... -Xcompiler "/EHsc /W3 /nologo /Od /Fdfish.dir\Debug\fish.pdb /FS /Zi /RTC1 /MDd /GR" ...

EDIT3: Some progress! With this simple change:

#add_compile_options("$<$<COMPILE_LANGUAGE:CUDA>:--compiler-options=/W4>")
add_compile_options("$<$<COMPILE_LANGUAGE:CUDA>:-Xcompiler=/W4>")

there are no more complaints! The confusing flags persist, but now read:

-Xcompiler "/EHsc /W4 /nologo /Od /Ffish.dir\Debug\fish.pdb
  /FS /Zi /RTC1 /MDd /GR"

ie. CMake has spotted the /W4 and overridden its default.

But despite scouring the CMake source, I've still yet to determine where all this is done.

EDIT4: MSVC has a hand in all this. The 'mystery' options are generated somewhat automatically by MSVC from the .vcxproj file. If I use the -Xcompile=/W4 approach, the project XML contains:

<CudaCompile>
  ...
  <AdditionalOptions>%(AdditionalOptions) -Xcompiler="/EHsc -Zi -Ob1"</AdditionalOptions>
  ...
  <Warning>W4</Warning>
 </CudaCompile>

But with the original, --compiler-options=/W4 I get:

<CudaCompile>
   <AdditionalOptions>%(AdditionalOptions) --compiler-options=/W4 -Xcompiler="/EHsc -Zi -Ob1"</AdditionalOptions>
   ...
</CudaCompile>

And this all sort of makes sense; with no Warning level specified, MSVC must default to W1, hence the complaint.

Also, there was a clue-with-hindsight: the build step had:

-Xcompiler="/EHsc -Zi -Ob0"
-Xcompiler "/EHsc /W1 /nologo /Od ..."

Note one has an equals and one does not implying they come from different places.

I think my investigation has gone far enough for now.

Does this answer your question? [How to set compiler options with CMake in Visual Studio 2017](https://stackoverflow.com/questions/45995784/how-to-set-compiler-options-with-cmake-in-visual-studio-2017) — Kevin, Jan 02 '20 at 14:31
Thanks @squareskittles. I'm not sure that link covers my situation (at least not at first reading). The issue I'm having seems related to having _two_ sets of conflicting compiler flags passed to MSVC through NVCC. But I'll read it again just to see if it provides any clues all the same. — Edd Inglis, Jan 02 '20 at 14:42
If the goal is to set the warning flag to `/W4`, and remove the `/W3` or `/W1`, you can remove the default flag from the `CMAKE_CXX_FLAGS_INIT` variable as suggested in the linked answer. — Kevin, Jan 02 '20 at 14:51
Thanks @squareskittles, I've updated my question with the sad news that this doesn't seem enough. — Edd Inglis, Jan 02 '20 at 15:12
Aside: I Googled `"EHsc /W3 /nologo /Od"` and it gave lots of hits (none that seem useful as yet), but this implies that it's a default set of flags coming from somewhere. But where?! — Edd Inglis, Jan 02 '20 at 15:14

score 6 · Accepted Answer · answered Jan 02 '20 at 17:03

A TL;DR summary of my own shaggy-dog-tale of a question:

I had:

add_compile_options("$<$<COMPILE_LANGUAGE:CUDA>:--compiler-options=/W4>")

but found:

add_compile_options("$<$<COMPILE_LANGUAGE:CUDA>:-Xcompiler=/W4>")

to be more appropriate.

Basically, the CMake/NVCC/MSVC pipeline understood that the -Xcompiler option overrode the default, but didn't realise that --compiler-options was an equivalent statement, resulting in a command line with ambiguous directives.

FWIW, I've raised a ticket with CMake: https://gitlab.kitware.com/cmake/cmake/issues/20164 — Edd Inglis, Jan 02 '20 at 17:33

score 0 · Answer 2 · answered May 18 '20 at 19:25

I had a similar issue (but with another flag) and used CMake's regular expressions to fix that. The code is like this.

foreach(flag_var
        CMAKE_CXX_FLAGS CMAKE_CXX_FLAGS_DEBUG CMAKE_CXX_FLAGS_RELEASE
        CMAKE_CXX_FLAGS_MINSIZEREL CMAKE_CXX_FLAGS_RELWITHDEBINFO)
    STRING (REGEX REPLACE "/W3" "/W4" ${flag_var} "${${flag_var}}")
endforeach(flag_var)

Confusing flags passed to MSVC through NVCC with CMake

2 Answers2

Linked