Automatically generate object file (linker) dependencies for C/C++ executables

Question

I am currently working on a flexible C/C++ build framework that I'll (hopefully) open source fairly soon. (see this question for some background).

I am using the below command to generate the #include file dependencies for source/header files.

gcc -M -MM -MF

Is there a way of cleverly inferring linker (.o file) dependencies for executables (unit tests + main executable for target platform in my case) using gcc/GNU utilties in a similar way to above? Currently the framework makes a whole lot of assumptions and is pretty dumb in determining these dependencies.

I have heard of one approach where the nm command can be used to come up with a list of undefined symbols in an object file. For example, running nm on an object file (compiled using gcc -c) comes up with something like this -

nm -o module.o

module.o:         U _undefinedSymbol1
module.o:         U _undefinedSymbol2
module.o:0000386f T _definedSymbol

One would then look for other object files where these undefined symbols are defined to come up with a list of object file dependencies required to successfully link the file.

Is this considered best practice in determining linker dependencies for executables? Are there any other ways of inferring these dependencies? Assume that all object files already exist (i.e. have already been compiled using gcc -c) when proposing your solution.

What would be the starting point for this approach? You initially have no object files, and therefore nothing to examine with `nm`... — Oliver Charlesworth, May 20 '12 at 02:53
Assume that all files have been compiled (i.e. just before the linking process) and their corresponding object files already exist. — thegreendroid, May 20 '12 at 02:54
In which case, there are no dependencies to resolve. If B is a dependency of A, it means that B is required in order to *create* A. — Oliver Charlesworth, May 20 '12 at 02:55
I am using gcc's *-c* option to compile all files but not link. — thegreendroid, May 20 '12 at 02:58
Yes, but you don't need one object file in order to create another object file. You need object files to create an executable (or shared library). So the top of your dependency tree will be the executable. If you don't know what the dependencies of the executable are, then you can't do anything. — Oliver Charlesworth, May 20 '12 at 03:00
Yes that's what I meant. i.e. figure out a way to determine object file dependencies in order to link executables. The executables are unit tests and a main executable for the whole project (that runs on an embedded target). — thegreendroid, May 20 '12 at 03:04
I want to build an executable called `foo`. What are its dependencies? — Oliver Charlesworth, May 20 '12 at 03:05
Let's say foo is a unit test called foo.c. So in order to successfully link it and create an executable, one would need foo.o and the unit it is testing (bar.o) and any other implicit dependencies of bar.o. — thegreendroid, May 20 '12 at 03:13
let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/11476/discussion-between-thegreendroid-and-oli-charlesworth) — thegreendroid, May 20 '12 at 03:13

score 8 · Accepted Answer · answered Sep 23 '12 at 07:30

If there are multiple executables (or even a single executable) that need different sets of dependencies, then the normal, classic way to handle that is to use a library — static .a or shared .so (or equivalent) — to hold the object files that can be used by more than one program, and to link the programs with that library. The linker automatically pulls the correct object files out of a static archive. The shared library process is a little different, but the net result is the same: the executable has the correct object files available to it at runtime.

For any program, there is at least one file unique to the program (normally, that's the file that contains the main() program). There may be a few files for that program. Those files are probably known about and can be listed easily. The ones that you might need depending on configuration and compilation options are probably shared between programs and are easily handled via the library mechanism.

You have to decide whether you want to use static or shared libraries. Creating shared libraries well is harder than creating static libraries. On the other hand, you can update a shared library and immediately affect all the programs that use it, whereas a static library can be changed but only programs that are relinked with the new library benefit from the changes.

score 5 · Answer 2 · answered May 29 '13 at 17:01

The following Python script can be used to collect and process the nm output for all object files in the current directory:

#! /usr/bin/env python

import collections
import os
import re
import subprocess

addr_re = r"(?P<address>[0-9a-f]{1,16})?"
code_re = r"(?P<code>[a-z])"
symbol_re = r"(?P<symbol>[a-z0-9_.$]+)"
nm_line_re = re.compile(r"\s+".join([addr_re, code_re, symbol_re]) + "\s*$",
                        re.I)

requires = collections.defaultdict(set)
provides = collections.defaultdict(set)

def get_symbols(fname):
    lines = subprocess.check_output(["nm", "-g", fname])
    for l in lines.splitlines():
        m = nm_line_re.match(l)
        symbol = m.group('symbol')
        if m.group('code') == 'U':
            requires[fname].add(symbol)
        else:
            provides[symbol].add(fname)

for dirpath, dirnames, filenames in os.walk("."):
    for f in filenames:
        if f.endswith(".o"):
            get_symbols(f)

def pick(symbols):
    # If several files provide a symbol, choose the one with the shortest name.
    best = None
    for s in symbols:
        if best is None or len(s) < len(best):
            best = s
    if len(symbols) > 1:
        best = "*" + best
    return best

for fname, symbols in requires.items():
    dependencies = set(pick(provides[s]) for s in symbols if s in provides)
    print fname + ': ' + ' '.join(sorted(dependencies))

The script searches the current directory and all subdirectories for .o files, calls nm for each file found and dissects the resulting output. Symbols which are undefined in one .o file and defined in another are interpreted as a dependency between the two files. Symbols defined nowhere (typically provided by external libraries) are ignored. Finally, the script prints a list of direct dependencies for all object files.

If a symbol is provided by several object files, this script arbitrarily assumes a dependency on the object file with the shortest file name (and marks the chosen file with a * in the output). This behaviour can be changed by modifying the function pick.

The script works for me on Linux and MacOS, I haven't tried any other operating systems and the script is only lightly tested.

Thanks for posting this. I stumbled here trying to answer a slightly different question--in an app that doesn't judiciously manage its dependencies, how can i figure out if anything really NEEDS to link against a particular library or object file. I'll likely have to modify your script, but it's always easier to build on top of something than create anew. — Chris Cleeland, Nov 30 '17 at 18:32
Also, at least with python 2.7, this didn't work without modification. In the loop over the return from os.walk() I had to join `dirpath` and `f` to get the filepath, and pass the filepath as argument to `get_symbols`. — Chris Cleeland, Nov 30 '17 at 19:02

score 4 · Answer 3 · answered Sep 25 '12 at 22:43

The nm utility reads object files (and archives, such as .a libraries) using libbfd. I'm thinking what you'll really want to do is process a database of the public symbols defined in the libraries you know about, and in the object files that a part of this project, so that as you generate each new object file you can look at the undefined symbols in it and determine which object -- plain or in a library -- you need to link to resolve the references. Essentially you're doing the same job as the linker, but sort of in reverse, so that you will which symbols you can locate.

If you're working with GCC, you can always look into the source packages for your 'binutils' to find the sources to nm, and even to ld if you want that. You certainly don't want to run nm and parse the output when it's just using libbfd under the hood, just call libbfd yourself.

Automatically generate object file (linker) dependencies for C/C++ executables

3 Answers3

Linked