Why is my C preprocessor ignoring these macros using spaces?

Question

I'm trying to build WRF 4.2, but have run into some errors due to a macro not being expanded by the preprocessor.

Essentially, WRF is build by csh scripts that configure and then compile the source. Part of that configuration process takes fixed format Fortran (.F files), and transforms them into free format Fortran files(.f90) which are then compiled into object files. The commands being run are generated according to the user's selected platform and compiler configuration, which for me is Darwin architecture, GNU (gfortran/gcc) compilers. I'm trying to compile the "serial" configuration. Here is the part of the output I'm concerned with:

sed -e "s/^\!.*'.*//" -e "s/^ *\!.*'.*//" module_mp_fast_sbm.F > module_mp_fast_sbm.G
cpp -P -nostdinc -xassembler-with-cpp -I/Users/isaacrowe/Documents/GitHub/WRF_ARW/WRF/inc -DEM_CORE=1 -DNMM_CORE=0 -DNMM_MAX_DIM=2600 -DDA_CORE=0 -DWRFPLUS=0 -DIWORDSIZE=4 -DDWORDSIZE=8 -DRWORDSIZE=4 -DLWORDSIZE=4 -DNONSTANDARD_SYSTEM_SUBR -DMACOS  -DWRF_USE_CLM  -DDM_PARALLEL -DSTUBMPI -DNETCDF -DLANDREAD_STUB=1 -DUSE_ALLOCATABLES -Dwrfmodel -DGRIB1 -DINTIO -DKEEP_INT_AROUND -DLIMIT_ARGS -DBUILD_RRTMG_FAST=0 -DBUILD_RRTMK=0 -DBUILD_SBM_FAST=1 -DSHOW_ALL_VARS_USED=0 -DCONFIG_BUF_LEN=65536 -DMAX_DOMAINS_F=21 -DMAX_HISTORY=25 -DNMM_NEST=0  -I. -traditional-cpp   module_mp_fast_sbm.G  > module_mp_fast_sbm.bb
/Users/isaacrowe/Documents/GitHub/WRF_ARW/WRF/tools/standard.exe module_mp_fast_sbm.bb | cpp -P -nostdinc -xassembler-with-cpp -traditional-cpp > module_mp_fast_sbm.f90
rm -f module_mp_fast_sbm.G module_mp_fast_sbm.bb
time gfortran -o module_mp_fast_sbm.o -c -O2 -ftree-vectorize -funroll-loops -w -ffree-form -ffree-line-length-none -fconvert=big-endian -frecord-marker=4   -I../dyn_em -I../dyn_nmm  -I/Users/isaacrowe/Documents/GitHub/WRF_ARW/WRF/external/esmf_time_f90  -I/Users/isaacrowe/Documents/GitHub/WRF_ARW/WRF/main -I/Users/isaacrowe/Documents/GitHub/WRF_ARW/WRF/external/io_netcdf -I/Users/isaacrowe/Documents/GitHub/WRF_ARW/WRF/external/io_int -I/Users/isaacrowe/Documents/GitHub/WRF_ARW/WRF/frame -I/Users/isaacrowe/Documents/GitHub/WRF_ARW/WRF/share -I/Users/isaacrowe/Documents/GitHub/WRF_ARW/WRF/phys -I/Users/isaacrowe/Documents/GitHub/WRF_ARW/WRF/wrftladj -I/Users/isaacrowe/Documents/GitHub/WRF_ARW/WRF/chem -I/Users/isaacrowe/Documents/GitHub/WRF_ARW/WRF/inc -I/usr/local/Cellar/netcdf/4.7.4/include    module_mp_fast_sbm.f90
module_mp_fast_sbm.f90:6065:4:

 6065 |     DM_BCAST_MACRO_R16 ( FAF1 )
      |    1
Error: Unclassifiable statement at (1)
        0.56 real         0.52 user         0.03 sys

What's happening here is that the Fortran file being compiled in the last step still has a unreplaced macro. Keep in mind that there are dozens of these Fortran files and this is the only one with an issue.

I looked at where the macro is defined in module_mp_fast_sbm.F and saw these definitions

#define DM_BCAST_MACRO_R4(A) CALL wrf_dm_bcast_bytes(A, size(A)*R4SIZE)
#define DM_BCAST_MACRO_R8(A) CALL wrf_dm_bcast_bytes(A, size(A)*R8SIZE)
#define DM_BCAST_MACRO_R16(A) CALL wrf_dm_bcast_bytes(A, size(A)*R16SIZE)

I also noticed that other uses of this same macro were replaced correctly, but those instances did not use the space between the macro name, the parenthesis and the parameter. This made me wonder if the spacing was making a difference. So I ran an experiment:

test.c:

#define DM_BCAST_MACRO_R4(A) CALL wrf_dm_bcast_bytes(A, size(A)*R4SIZE)

DM_BCAST_MACRO_R4(3);

// The style used by WRF
DM_BCAST_MACRO_R4 ( 3 );

// Just curious
DM_BCAST_MACRO_R4( 3);

DM_BCAST_MACRO_R4 (3);

First I used the command as it was used in the WRF script:

cat test.c | cpp -P -nostdinc -xassembler-with-cpp -traditional-cpp > output_traditional.txt

And I got this output:



CALL wrf_dm_bcast_bytes(3, size(3)*R4SIZE);

// The style used by WRF
DM_BCAST_MACRO_R4 ( 3 );

// Just curious
CALL wrf_dm_bcast_bytes( 3, size( 3)*R4SIZE);

DM_BCAST_MACRO_R4 (3);

I thought maybe the flags WRF was using could be causing the issue, so I tried this command:

cat test.c | cpp -P  > output_noflags.txt

And got this result:

CALL wrf_dm_bcast_bytes(3, size(3)*R4SIZE);

// The style used by WRF
DM_BCAST_MACRO_R4 ( 3 );

// Just curious
CALL wrf_dm_bcast_bytes( 3, size( 3)*R4SIZE);

DM_BCAST_MACRO_R4 (3);

The output of cpp -v

Apple clang version 11.0.3 (clang-1103.0.32.62)
Target: x86_64-apple-darwin19.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -cc1 -triple x86_64-apple-macosx10.15.0 -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -E -disable-free -disable-llvm-verifier -discard-value-names -main-file-name - -mrelocation-model pic -pic-level 2 -mthread-model posix -mframe-pointer=all -fno-strict-return -masm-verbose -munwind-tables -target-sdk-version=10.15.4 -target-cpu penryn -dwarf-column-info -debugger-tuning=lldb -target-linker-version 556.6 -v -resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/11.0.3 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -I /usr/include -I/usr/local/include -internal-isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/local/include -internal-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/11.0.3/include -internal-externc-isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -internal-externc-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include -Wno-objc-signed-char-bool-implicit-int-conversion -Wno-extra-semi-stmt -Wno-quoted-include-in-framework-header -fdebug-compilation-dir /Users/isaacrowe/Desktop -ferror-limit 19 -fmessage-length 270 -stack-protector 1 -fstack-check -mdarwin-stkchk-strong-link -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fobjc-runtime=macosx-10.15.0 -fmax-type-align=16 -fdiagnostics-show-option -fcolor-diagnostics -traditional-cpp -o - -x c -
clang -cc1 version 11.0.3 (clang-1103.0.32.62) default target x86_64-apple-darwin19.5.0
ignoring nonexistent directory "/usr/include"
ignoring nonexistent directory "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/local/include"
ignoring nonexistent directory "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/Library/Frameworks"
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/11.0.3/include
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks (framework directory)
End of search list.

From what I read here, the macro definition cannot have a space before the open parenthesis, but the a use of the macro may have a space before the parenthesis. So why does my preprocessor not make the replacement?

@NateEldredge the second usage of the macro (the one under the comment `// The style used by WRF`) doesn't get replaced in either output. The expected behavior is that the preprocessor will replace it even with the space before the left parenthesis. — irowe, Jun 05 '20 at 14:33
On [godbolt](https://godbolt.org/z/5ULdD3), I see the "incorrect" behavior when using `-traditional-cpp`, but not without it. Are you sure you didn't get your tests mixed up? Or is it possible your cpp is getting default options from somewhere? — Nate Eldredge, Jun 05 '20 at 14:46
Oh, https://stackoverflow.com/questions/43967458/difference-between-cpp-and-gcc-e suggests that `-traditional-cpp` is the default when running `cpp` by itself, but not when doing `cc -E` which is what godbolt does. At least for gcc, and clang may be the same. But gcc's `cpp -traditional-cpp` doesn't have this behavior, so that part may be a clang bug, or different interpretation of "traditional". — Nate Eldredge, Jun 05 '20 at 14:49
So perhaps the real question is how do you tell clang's `cpp` to not use `-traditional-cpp`. I don't have it installed to test, but you could try something like `cpp -std=c90`. — Nate Eldredge, Jun 05 '20 at 15:14
Thanks for digging into that. On my machine, gcc and cpp look like they have the same behavior with `traditional-cpp` so looks like maybe it is expected behavior. Let me check to see what I can do do run cpp without that flag. The next step would be to figure out how to get the WRF script to generate the right flags for me. — irowe, Jun 05 '20 at 15:40
May be a known bug: https://bugs.llvm.org/show_bug.cgi?id=18011 — Nate Eldredge, Jun 05 '20 at 15:44
Keep in mind that on some Apple systems, the `gcc` command is really clang, not the GNU compiler. — Nate Eldredge, Jun 05 '20 at 15:48
Wow, I didn't know that. Turns out `gcc` on my system was actually clang. I ran my `gcc-9` that I installed with Homebrew and got the output I was looking for (all instances replaced). Also, it's really depressing to see that that bug is so ancient with no activity. — irowe, Jun 05 '20 at 15:58
Glad you got it resolved. Would you like to write up a quick answer? — Nate Eldredge, Jun 05 '20 at 16:04
You've put in all the legwork, I'll let you get the rep if you want to write one up. Otherwise I can write it up myself. — irowe, Jun 05 '20 at 16:07

score 1 · Accepted Answer · answered Jun 05 '20 at 16:23

This behavior of clang's cpp when using -traditional-cpp has been reported back in 2013, with no followup to date. It seems like it's probably a bug, since gcc's preprocessor doesn't do it (example), and it looks like clang prior to version 3.2 didn't do it either. But the the intended semantics of -traditional-cpp are not precisely defined anywhere that I know of, and the clang maintainers didn't respond to the bug report, so it's hard to know for sure. It's also possible that they just don't care about -traditional-cpp and consider it obsolete.

It looks like -traditional-cpp is the default for your installation when running cpp standalone. I don't know exactly how to turn it off, but -std=c90 might do it. Note that clang -E also avoids traditional mode and seems to behave correctly.

You can also use gcc's preprocessor instead which doesn't have this "bug". As you found, on your system the gcc command is actually clang, but you can install the real gcc from Homebrew.

I [contributed a fix](https://github.com/wrf-model/WRF/pull/1221) to WRF to change the macro spacing in order to work around what is indeed a bug in the clang preprocessor. Thanks for the great answer. — irowe, Oct 26 '22 at 15:32

Why is my C preprocessor ignoring these macros using spaces?

1 Answers1