How to identify "dead c-files" in a large C-project?

Question

Note: before marking this as a duplicate, please read the whole question,
especially chapter 5 :-)

1. Use case

Suppose you have a large C-project with numerous .h and .c files. But most of those files are never used. This is typically the case for embedded software projects. The drivers for each peripheral of your microcontroller are present somewhere in the project, but you typically only use a few of them. If - for example - I use the usb-driver and the uart-driver, I will import the corresponding .h files in my main.c:

#include "stm32f7xx_usb.h"
#include "stm32f7xx_uart.h"

Okay ... I have overly simplified the use case, but you get the point.

2. Dead files

Let's agree on the following quick-and-dirty definition for a dead file:

A dead .c file is a file which functions or data will never be reached, because the corresponding .h file is not imported.

I know, it's quite an unclean definition. There is not always a one-to-one relationship between .c and .h files. So perhaps I should say:

A .c file is dead if none of its functions/data is exposed through one or more imported .h files.

Usually in embedded software, we enclose such import statements within an #ifdef block:

#ifdef USE_USB
    #include "stm32f7xx_usb.h"
#endif

This means that the state "dead" or "alive" can be changed at compile-time, depending on which preprocessor macros you inject when compiling. Therefore, let us assume that all preprocessor macros injected at compile-time are known. They can simply be found in the makefile.

3. Ultimate goal:
distinguish dead files before compilation

Sometimes one of the dead files has an error. It is very frustrating when that error freezes the whole compilation process - especially because the file won't be used anyhow. Knowing somehow which files are dead before compilation, can be very useful.
But I have honestly no idea what tool can be used for this. And how.

PS: of course some kind of precompiler or parser should run anyhow. But that's okay.

4. Less favorable goal:
distinguish dead files after compilation

I think I'm getting closer to this goal, but I'm not there yet. I'll show where I am right now.

Passing on the arguments -ffunction-sections and -fdata-sections to the gcc compiler, results in cleanly separated function- and datablocks in the generated object files.
Next, I pass the arguments -Wl,--gc-sections and -Wl,--print-gc-sections to the gcc linker. The linker kicks out all unreacheable data and functions, resulting in a much smaller binary file. Also, the linker prints out what he's throwing away:

...
ld.exe: Removing unused section '.text.HAL_UART_Receive_IT' in file 'build\Drivers\..\stm32f7xx_hal_uart.o'
ld.exe: Removing unused section '.text.HAL_UART_Transmit_DMA' in file 'build\Drivers\..\stm32f7xx_hal_uart.o'
ld.exe: Removing unused section '.text.HAL_UART_Receive_DMA' in file 'build\Drivers\..\stm32f7xx_hal_uart.o'
ld.exe: Removing unused section '.text.HAL_UART_DMAPause' in file 'build\Drivers\..\stm32f7xx_hal_uart.o'
ld.exe: Removing unused section '.text.HAL_UART_DMAResume' in file 'build\Drivers\..\stm32f7xx_hal_uart.o'
ld.exe: Removing unused section '.text.HAL_UART_DMAStop' in file 'build\Drivers\..\stm32f7xx_hal_uart.o'
ld.exe: Removing unused section '.text.HAL_UART_IRQHandler' in file 'build\Drivers\..\stm32f7xx_hal_uart.o'
ld.exe: Removing unused section '.text.UART_DMATransmitCplt' in file 'build\Drivers\..\stm32f7xx_hal_uart.o'
...

This is great, but it doesn't tell me if the whole file is dead. Perhaps a small part of the file is still used somewhere - a single function maybe - such that the file should be considered "alive".
For that reason, I cannot agree with the answer of @Mike Kinghan on this post: gcc linker get list of unused objects

I heard that the state of a file (dead or alive) can be found in the output.map file. I've opened the output.map file, but quickly gave up reading it. It is 30.342 lines! If you believe it can be found in the output.map file, please tell me where to look :-).

5. Why I need to know the dead files

The exact reasons why will lead us too far here. It would only distract the focus. Just assume it's important ;-)

6. Why this is not a duplicate question

I know there are several questions on StackOverflow related to "dead code" dectection in C-programs. However, I'm not interested in dead code. I'm interested in dead files. Of course these are related topics, but they are not duplicates.
After all, files that are alive can still contain a lot of dead code. As long as there's at least one function or variable exposed through an imported .h file, I would consider the file as "alive".

Even this question about unused object files (gcc linker get list of unused objects) is not a duplicate. My ultimate goal is to detect the state of a file (dead or alive) BEFORE compilation takes place.

Thank you very much @cleblanc, @zwol and @user2162550 for helping me out! I've focused a bit more on the answer of @zwol, because it avoids compilation alltogether. I'll show below what I got.

> First proposed solution

This solution comes from @zwol. I issue the following command in my terminal:

arm-none-eabi-gcc -H -fsyntax-only main.c
    -IX:\source\Inc
    -IX:\source\Drivers\CMSIS\RTOS\Template
    -IX:\source\Drivers\STM32F7xx_HAL_Driver\Inc
    -IX:\source\Drivers\CMSIS\Include
    -IX:\source\Drivers\STM32F7xx_HAL_Driver\Inc\Legacy
    -IX:\source\Drivers\CMSIS\Device\ST\STM32F7xx\Include

This command should be on a single line, but for clarity I've inserted some linebreaks.

I also tried to add the following preprocessor macros to the command:

    -DSTM32F746xx
    -DARM_MATH_CM7
    -D__weak="__attribute__((weak))"
    -D__packed="__attribute__((__packed__))"
    -DUSE_HAL_DRIVER

But it causes the command to fail. So I had no choice but to insert these macros directly into the main.c file at the top:

#define STM32F746xx
#define ARM_MATH_CM7
#define __weak __attribute__((weak))
#define __packed __attribute__((__packed__)
#define USE_HAL_DRIVER

Now the command works.

I get quite a lot of output. Please note that X:\ is in fact the location of the C-project root folder on my harddrive. I've substituted that root folder path by X:\ for clarity.

. X:\source\Inc/main.h
. X:\source\Drivers\STM32F7xx_HAL_Driver\Inc/stm32f7xx_hal.h
.. X:\source\Inc/stm32f7xx_hal_conf.h
... X:\source\Inc/main.h
... X:\source\Drivers\STM32F7xx_HAL_Driver\Inc/stm32f7xx_hal_rcc.h
.... X:\source\Drivers\STM32F7xx_HAL_Driver\Inc/stm32f7xx_hal_def.h
..... X:\source\Drivers\CMSIS\Device\ST\STM32F7xx\Include/stm32f7xx.h
...... X:\source\Drivers\CMSIS\Device\ST\STM32F7xx\Include/stm32f746xx.h
....... X:\source\Drivers\CMSIS\Include/core_cm7.h
........ c:\gnu_arm_embedded_toolchain\lib\gcc\arm-none-eabi\6.3.1\include\stdint.h
......... c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\stdint.h
.......... c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\machine\_default_types.h
........... c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\sys\features.h
............ c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\_newlib_version.h
.......... c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\sys\_intsup.h
.......... c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\sys\_stdint.h
........ X:\source\Drivers\CMSIS\Include/core_cmInstr.h
......... X:\source\Drivers\CMSIS\Include/cmsis_gcc.h
........ X:\source\Drivers\CMSIS\Include/core_cmFunc.h
........ X:\source\Drivers\CMSIS\Include/core_cmSimd.h
....... X:\source\Drivers\CMSIS\Device\ST\STM32F7xx\Include/system_stm32f7xx.h
...... X:\source\Drivers\STM32F7xx_HAL_Driver\Inc/stm32f7xx_hal.h
..... X:\source\Drivers\STM32F7xx_HAL_Driver\Inc/Legacy/stm32_hal_legacy.h
..... c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\stdio.h
...... c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\_ansi.h
....... c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\newlib.h
....... c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\sys\config.h
........ c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\machine\ieeefp.h
...... c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\sys\cdefs.h
....... c:\gnu_arm_embedded_toolchain\lib\gcc\arm-none-eabi\6.3.1\include\stddef.h
...... c:\gnu_arm_embedded_toolchain\lib\gcc\arm-none-eabi\6.3.1\include\stddef.h
...... c:\gnu_arm_embedded_toolchain\lib\gcc\arm-none-eabi\6.3.1\include\stdarg.h
...... c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\sys\reent.h
....... c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\_ansi.h
....... c:\gnu_arm_embedded_toolchain\lib\gcc\arm-none-eabi\6.3.1\include\stddef.h
....... c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\sys\_types.h
........ c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\machine\_types.h
........ c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\sys\lock.h
........ c:\gnu_arm_embedded_toolchain\lib\gcc\arm-none-eabi\6.3.1\include\stddef.h
...... c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\sys\types.h
....... c:\gnu_arm_embedded_toolchain\lib\gcc\arm-none-eabi\6.3.1\include\stddef.h
....... c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\machine\endian.h
........ c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\machine\_endian.h
....... c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\sys\select.h
........ c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\sys\_sigset.h
........ c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\sys\_timeval.h
........ c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\sys\timespec.h
......... c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\sys\_timespec.h
....... c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\sys\_pthreadtypes.h
....... c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\machine\types.h
...... c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\sys\stdio.h
.... X:\source\Drivers\STM32F7xx_HAL_Driver\Inc/stm32f7xx_hal_rcc_ex.h
... X:\source\Drivers\STM32F7xx_HAL_Driver\Inc/stm32f7xx_hal_gpio.h
.... X:\source\Drivers\STM32F7xx_HAL_Driver\Inc/stm32f7xx_hal_gpio_ex.h
... X:\source\Drivers\STM32F7xx_HAL_Driver\Inc/stm32f7xx_hal_dma.h
.... X:\source\Drivers\STM32F7xx_HAL_Driver\Inc/stm32f7xx_hal_dma_ex.h
... X:\source\Drivers\STM32F7xx_HAL_Driver\Inc/stm32f7xx_hal_cortex.h
... X:\source\Drivers\STM32F7xx_HAL_Driver\Inc/stm32f7xx_hal_eth.h
... X:\source\Drivers\STM32F7xx_HAL_Driver\Inc/stm32f7xx_hal_flash.h
.... X:\source\Drivers\STM32F7xx_HAL_Driver\Inc/stm32f7xx_hal_flash_ex.h
... X:\source\Drivers\STM32F7xx_HAL_Driver\Inc/stm32f7xx_hal_i2c.h
.... X:\source\Drivers\STM32F7xx_HAL_Driver\Inc/stm32f7xx_hal_i2c_ex.h
... X:\source\Drivers\STM32F7xx_HAL_Driver\Inc/stm32f7xx_hal_pwr.h
.... X:\source\Drivers\STM32F7xx_HAL_Driver\Inc/stm32f7xx_hal_pwr_ex.h
... X:\source\Drivers\STM32F7xx_HAL_Driver\Inc/stm32f7xx_hal_uart.h
.... X:\source\Drivers\STM32F7xx_HAL_Driver\Inc/stm32f7xx_hal_uart_ex.h
... X:\source\Drivers\STM32F7xx_HAL_Driver\Inc/stm32f7xx_hal_pcd.h
.... X:\source\Drivers\STM32F7xx_HAL_Driver\Inc/stm32f7xx_ll_usb.h
.... X:\source\Drivers\STM32F7xx_HAL_Driver\Inc/stm32f7xx_hal_pcd_ex.h

Multiple include guards may be useful for:
X:\source\Drivers\CMSIS\Include/core_cm7.h
X:\source\Drivers\CMSIS\Include/core_cmFunc.h
X:\source\Drivers\CMSIS\Include/core_cmInstr.h
X:\source\Drivers\CMSIS\Include/core_cmSimd.h
c:\gnu_arm_embedded_toolchain\arm-none-eabi\include\machine\_endian.h

I will need some more time to wrap my head around this result. But it definitely feels like getting closer to what I eventually need. Thank you @zwol!

PS: If you know why I cannot inject preprocessor macros into the command, please tell me ;-)

BTW, according to your section #2, `A .c file is dead if none of its functions/data is exposed through one or more imported .h files` we can deduct that `c` file that some of it's functions\data are used in another `c` file using forward declaration is considered dead, but I think that is not what you meant — izac89, Apr 11 '18 at 17:57
Yeah, that's not what I mean. Well, I must admit I couldn't find the inspiration to properly define a "dead c-file". But I hope anyone reading the full question, somehow gets the point what I mean by "dead". If you have a good definition in mind, please share with me. It would be helpful :-) — K.Mulier, Apr 11 '18 at 18:00
I'd probably just build all of the object files, then try removing them one by one to see when the linker complained, and automating that approach if there were too many to do by hand; that's the crude-but-effective approach. — Beta, Apr 11 '18 at 21:39
Regarding the insertion of preprocessor macros, this is an issue with CMD.EXE/shell/Makefile quotation, not specific to what you're trying to do here. You should ask a separate question about that, in which you follow the [MCVE](https://stackoverflow.com/help/mcve) instructions to demonstrate the problem. We will also need to know exactly which command-line environment you are using, and we will need to see the _complete and unedited_ text of the error messages. — zwol, Apr 12 '18 at 00:13

score 6 · Answer 1 · answered Apr 11 '18 at 18:24

If you try NOT passing on the arguments -ffunction-sections and -fdata-sections to the gcc compiler, the linker will be forced to link the entire object if any source or data is used. Then pass the arguments -Wl,--gc-sections and -Wl,--print-gc-sections to the gcc linker and the linker should tell you which object files were completely unused.

score 4 · Answer 2 · answered Apr 11 '18 at 18:29

If you are using GCC specifically, gcc -H -fsyntax-only main.c will print a list of every file transitively included by main.c, instead of compiling it. (It will still parse the file and report syntax errors. That's unavoidable.)

You can do this for every file that definitely must be included in your project, and then crunch the output lists to determine the full set of "live" C source files. I can't help you with that part because you didn't explain exactly how the header files are related to the source files.

You will probably need to weed out headers belonging to whatever subset of the C standard library you have available.

Thank you very much. I've applied your solution, and written down the results in `First proposed solution`. — K.Mulier, Apr 11 '18 at 19:05

izac89 · Answer 3 · 2018-04-11T18:49:25.503

1

If using compilation flags -ffunction-sections and -fdata-sections is a must in your project, then for GCC you can use the following post link approach (else, cleblanc's answer is the easiest solution)-

For every object file listed in Discarded input sections inside the map file, if this object file is not listed in text, data and bss sections, (i.e only in Discarded input sections, debug symbols section, Cross Reference Table, ...) it means that this file is "dead" according to your definition, this file was compiled, linked, but then was found "dead" and totally discarded.

edited Apr 11 '18 at 18:49

answered Apr 11 '18 at 18:32

izac89

3,790
7
30
46

Why not use `--print-gc-sections` – technosaurus Apr 11 '18 at 21:03
@technosaurus because flag `--print-gc-sections` with compilation flags `-ffunction-sections` and `-fdata-sections` will cause objects that only parts of their functions\data (which becomes sections) are discarded to be printed, which means that `live` file was printed, along with `dead` files which got all of their functions\data discarded – izac89 Apr 12 '18 at 00:46