Can I display the progress of a build using SCons?

Question

I have a simple build system for a project that I've put together using SCons. The project has quite a few source files, and I thought it would be user-friendly to display some sort of progress information as the build is progressing. SCons provides construction variables like CXXCOMSTR that I can override to control what gets displayed to the terminal during each of the build steps. For instance, instead of seeing something like:

gcc file.c -o file.o

It would be nice to have something like:

[1/10] Compiling file.c: `gcc file.c -o file.o`

Where the [1/10] specifies that this is the first of ten targets that are being updated during this build. Is there any way to access this information so that I can generate messages like this? It seems like I would need to know the total number of targets that are being updated (which is based upon the dependency scanning that SCons does) and some way of enumerating each one. I know that similar behavior is similar with other build systems like CMake and waf, but haven't come across anything in SCons's (expansive) documentation.

Answer 1

This can be done with the Progress() SCons function.

There are more examples in the SCons man page, just search for "Progress".

You can either provide a function to be called, a string that will be displayed periodically, or a list of strings that will be displayed in a rotating fashion. You can get access to the current target being built, but I dont think its possible to know the percentage complete.

Answer 2

I was able to come up with a fairly decent workaround that allows you to display build progress as a percentage. Basically how the following code works is this:

• If a target has never been built before, it will display [??%] next to each target
• During the first time it is being built, it will cache away the total nodes processed for that target in .scons_progress
• Next time you run a build, it will read from the file on start and get the total nodes from that target last time and use it to display a percentage (i.e. [50%]).
• If the number of nodes ever exceeds the cached total, it will revert back to [??%] and update .scons_progress
• Over time as your SConstruct "stabilizes" (i.e. you aren't making changes all the time), these cache values should get more and more accurate.
• Updating the cache is going to slow down your build (slightly per node) so you might want to set the interval to 5 so that it only updates the cache every 5 nodes

Here is the code:

In custom_utils.py under site_scons/:

PROGRESS = {
    "env": None,
    "nodes": 0,
    "interval": 1,
    "total": 0,
    "cache": {},
    "cache_key": "_",
    "cache_name": ".scons_progress",
    "cache_update": False
}

def get_progress_total():
    return PROGRESS["cache"].get(PROGRESS["cache_key"], 0)

def update_progress_cache():
    PROGRESS["cache"][PROGRESS["cache_key"]] = PROGRESS["nodes"]
    with open(PROGRESS["cache_name"], "w") as f:
        f.write(json.dumps(PROGRESS["cache"]))

def load_progress_cache():
    try:
        with open(PROGRESS["cache_name"]) as f:
            PROGRESS["cache"] = json.load(f)

            cache_key = "_".join(COMMAND_LINE_TARGETS)
            if cache_key:
                PROGRESS["cache_key"] = cache_key
    except IOError as e:
        pass

def progress_function(node):
    PROGRESS["nodes"] += PROGRESS["interval"]

    #Only update cached environment if it is not None
    if node.env:
        PROGRESS["env"] = node.env
        PROGRESS["env"].Replace(PROGRESS = "[??%]")

    #Update display message
    if PROGRESS["nodes"] <= PROGRESS["total"]:
        PROGRESS["env"].Replace(PROGRESS = "[%d%%]" % (100*PROGRESS["nodes"]/PROGRESS["total"]))
    #If current node is more than total, we need to update the cache
    elif not PROGRESS["cache_update"]:
        PROGRESS["cache_update"] = True
        PROGRESS["env"].Replace(PROGRESS = "[??%]")
    #If cache flag is on, update cache file
    else:
        update_progress_cache()

def progress_settings(env, interval):
    load_progress_cache()
    PROGRESS["interval"] = interval
    PROGRESS["env"]      = env
    PROGRESS["total"]    = get_progress_total()
    Progress(progress_function, interval=PROGRESS["interval"])

In SConstruct:

if GetOption("progress"):
    custom_utils.progress_settings(env, 5)

Sample output:

user@linuxbox:~/project1$ scons -j4 --progress
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
[??%] Compiling (shared): src/ExpressRuntimeException.os
[??%] Compiling (shared): src/Application.os
[??%] Compiling (shared): src/Callback.os
[??%] Compiling (shared): src/Response.os
[??%] Compiling (shared): src/Router.os
[??%] Compiling (shared): src/Route.os
[??%] Compiling (shared): src/Regex.os
[??%] Compiling (shared): src/tools.os
[??%] Linking   (shared): libexpresscpp.so
scons: done building targets.

user@linuxbox:~/project1$ scons -c
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Cleaning targets ...
Removed src/ExpressRuntimeException.os
Removed src/Application.os
Removed src/Callback.os
Removed src/Response.os
Removed src/Router.os
Removed src/Route.os
Removed src/Regex.os
Removed src/tools.os
Removed libexpresscpp.so
scons: done cleaning targets.

user@linuxbox:~/project1$ scons -j4 --progress
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
[14%] Compiling (shared): src/ExpressRuntimeException.os
[51%] Compiling (shared): src/Application.os
[59%] Compiling (shared): src/Callback.os
[66%] Compiling (shared): src/Response.os
[74%] Compiling (shared): src/Router.os
[81%] Compiling (shared): src/Route.os
[88%] Compiling (shared): src/Regex.os
[96%] Compiling (shared): src/tools.os
[100%] Linking   (shared): libexpresscpp.so
scons: done building targets.

Answer 3

I liked RPGilespie's answer a lot and played around with it for our software package that has ~4500 nodes. I did a little tweaking though and thought I'd post it in case it helps anyone. The main differences are:

1.) I added a command to run at the end of the job to write the total node count to a file based on this post. This saves having to continually hit the file system in the progress function.

2.) Due to 1.) it updates the file every time scons is run (not just when it senses the number of nodes has exceeded the previous maximum). This is useful if you are working on a single part of the tree (i.e. fewer nodes) and are constantly rebuilding just that part.

3.) I have it put the carriage return before the screen update so it can write it straight from the progress_function as opposed to setting a variable for later. This has the added benefit of showing the counter updating while it scans through a mostly-built tree at the beginning.

As a side note, I set the update interval to just 1 here. The reason is because I noticed sometimes the number of nodes associated with a single build command was less than the interval causing it not to print the counter for that line. With the above change to how the counter file is written, I did not see any noticeable slowdown due to this. YMMV.

I put the following at the end of my SConstruct file:

screen = open('/dev/tty', 'w')
node_count = 0
node_count_max = 0
node_count_interval = 1
node_count_fname = str(env.Dir('#')) + '/.scons_node_count'

def progress_function(node):
    global node_count, node_count_max, node_count_interval, node_count_fname
    node_count += node_count_interval
    if node_count > node_count_max: node_count_max = 0
    if node_count_max>0 :
        screen.write('\r[%3d%%] ' % (node_count*100/node_count_max))
        screen.flush()

def progress_finish(target, source, env):
    global node_count
    with open(node_count_fname, 'w') as f: f.write('%d\n' % node_count)

try:
    with open(node_count_fname) as f: node_count_max = int(f.readline())
except: pass
Progress(progress_function, interval=node_count_interval)

progress_finish_command = Command('progress_finish', [], progress_finish)
Depends(progress_finish_command, BUILD_TARGETS)
if 'progress_finish' not in BUILD_TARGETS:     
    BUILD_TARGETS.append('progress_finish')

Example output is:

[  0%] Installing [/Users/davidl/HallD/builds/sim-recon/Darwin_macosx10.11-x86_64-llvm8.0.0/bin/MakeEventWriterROOT.pl] 
[  0%] Installing [/Users/davidl/HallD/builds/sim-recon/Darwin_macosx10.11-x86_64-llvm8.0.0/bin/MakeReactionPlugin.pl] 
[  0%] Compiling  [programs/Simulation/genr8/genkin.c] 
[  0%] Compiling  [programs/Simulation/genr8/genr8.c] 
[  3%] Compiling  [programs/Utilities/hddm/hddm-cpp.cpp] 
[  3%] Compiling  [programs/Utilities/hddm/XString.cpp] 
[  3%] Compiling  [programs/Utilities/hddm/XParsers.cpp] 
[  3%] Compiling  [programs/Utilities/hddm/md5.c] 
[  4%] Compiling  [external/xstream/src/base64.cpp] 
[  4%] Compiling  [external/xstream/src/bz.cpp] 
[  4%] Compiling  [external/xstream/src/common.cpp] 
[  4%] Compiling  [external/xstream/src/dater.cpp] 
[  4%] Linking    [.Darwin_macosx10.11-x86_64-llvm8.0.0/programs/Simulation/genr8/genr8] 
[  4%] Installing [/Users/davidl/HallD/builds/sim-recon/Darwin_macosx10.11-x86_64-llvm8.0.0/bin/genr8] 
[  4%] Compiling  [external/xstream/src/debug.cpp] 
[  4%] Compiling  [external/xstream/src/digest.cpp]
 ...

For completeness, here are my COMSTR definitions:

env.Replace(  CCCOMSTR        = "Compiling  [$SOURCE]",
              CXXCOMSTR       = "Compiling  [$SOURCE]",
              FORTRANPPCOMSTR = "Compiling  [$SOURCE]",
              FORTRANCOMSTR   = "Compiling  [$SOURCE]",
              SHCCCOMSTR      = "Compiling  [$SOURCE]",
              SHCXXCOMSTR     = "Compiling  [$SOURCE]",
              LINKCOMSTR      = "Linking    [$TARGET]",
              SHLINKCOMSTR    = "Linking    [$TARGET]",
              INSTALLSTR      = "Installing [$TARGET]",
              ARCOMSTR        = "Archiving  [$TARGET]",
              RANLIBCOMSTR    = "Ranlib     [$TARGET]")