How preprocessor directives works in C?

Question

I am going through book [Let us C-by Yashwant Kanetkar ], here it stated:

When we compile a program, before the source code passes to the compiler, it is examined by the C preprocessor for any macro definition. When it sees the #define directive, it goes through the entire program in search of macro templates; wherever it finds one, it replaces the macro template with the appropriate macro expansion. Only after this procedure has been completed, is the program handled over to the compiler.

My question is that, before the program is passed to compiler, how can Preprocessor program is able to read the TOKENS corresponding to the macro templates? Is preprocessor program also able to divide the program into TOKENS.

Answer 1

That description is confusing (so I won't recommend that book; read instead the K&R The C Programming Language book). The preprocessor does not go through the entire program, it has previously processed some input. Only past preprocessed input matters for the behavior of the preprocessor (in other words, the preprocessor is a single-pass mechanism).

Read wikipage on C preprocessor, then read documentation of GNU cpp and other documentation on preprocessor, and the wikibook chapter on C programming/Preprocessor.

In current C compilers (for performance reasons) the preprocessor is no longer a separate program, it is part of the compiler itself. For recent GCC look into libcpp/ (its preprocessor library, internal to the compiler).

If using the GCC compiler, you can get the preprocessed form of your source code file csource.c by running gcc -C -E csource.c > csource.i then looking inside the generated preprocessed form csource.i (e.g. with a pager or an editor).

^{(I strongly recommend doing that once in a while; you'll learn a lot; and yes, you could be surprised by the amount of code pulled by a usual #include <stdio.h> directive)}

I believe your book is explaining wrongly. The preprocessor handles every preprocessing directive. When it encounters a #define it stores in some preprocessor symbol table the definition of that symbol. When it encounters after that #define an occurrence of that preprocessor symbol, it does the appropriate substitution.

Answer 2

In book K & R The C Programming Language.

Page No: 88

C provides certain language facilities by means of a preprocessor, which is conceptually a separate first step in compilation.

In book Compiler Principles, Techniques and Tools by Aho, Lam, Sethi and Ullman

Page No. 3

The task of collecting the source program is sometimes entrusted to a separate program, called a preprocessor. The preprocessor may also expand shorthands, called macros, into source language statements. The modified source program is then fed to compiler.

In GCC GNU Documentation

The C preprocessor is a macro processor that is used automatically by the C compiler to transform your program before actual compilation.

Andn read this too.

So from these three official sources, one can say that the Preprocessor is a separate program run by Compiler. So in book Let Us C by Yashwant P Kanetkar that Preprocessor is a program that processes before the compiler as its name suggests is no wrong, and the expanded code can be seen in file.i.

---

Now let's come to your question,

In book K & R The C Programming Language.

Page No: 89

Substitution are made only for tokens and do not take place within quoted strings.

and as Basile told in his answer that

In current C compilers (for performance reasons) the preprocessor is no longer a separate program, it is part of the compiler itself.

and compiling is a long process that passes through several phases, Preprocessor actually comes after the program is converted in tokens, but as sources says that it is the process of before compilation that means it is done before any kind of intermediate code generation, and yes, breaking program into tokens is the first step of compiler before any intermediate code generation.