How to optimize the startup time of an SCons script?

Question

I have an SCons script that takes around 10 seconds just to find out that nothing needs to be rebuild, which feels awfully long for what is essentially a rather small project. Reading the SConscript itself takes just a second or two, most of the time is spend at the:

scons: Building targets ...

step.

How can I find out what exactly scons is doing at this point? And what other general advise can be given on writing fast SCons scripts?

Answer 1

(Stolen directly from http://www.scons.org/wiki/GoFastButton)

The command 'scons --max-drift=1 --implicit-deps-unchanged' will execute your build as fast as possible.

OR:

• env.Decider('MD5-timestamp'): as of SCons 0.98, you can set the Decider function on an environment. MD5-timestamp says if the timestamp matches, don't bother re-MD5ing the file. This can give huge speedups. See the man page for info.
• --max-drift: By default SCons will calculate the MD5 checksum of every source file in your build each time it is run, and will only cache the checksum after the file is 2 days old. This default of 2 days is to protect from clock skew from NFS or revision control systems. You can tweak this delay using --max-drift=SECONDS where SECONDS is some number of seconds. Decreasing SECONDS can improve build speed by eliminating superfluous MD5 checksum calculations. Instead of specifying this on the command line each run, you can set this option inside your SConstruct or SConscript file using "SetOption('max_drift', SECONDS)".
• --implicit-deps-unchanged: By default SCons will rescan all source files for implicit dependencies (e.g. C/C++ header #includes), which can be an expensive process. You can tell SCons to cache the implicit dependencies between invocations using the --implicit-deps-unchanged. By using this options you are making a promise to SCons that you didn't change any of the implicit dependencies since it was last run. If you do change the implicit dependencies, then using --implicit-deps-changed will cause them to be rescaned and cached. (You cannot set this option from within the SConstruct or SConscript files.)
• --implicit-cache: This option tells SCons to intelligently cache implicit dependencies. It attempts to determine if the implicit dependencies have changed since the last build, and if so it will recalculate them. This is usually slower than using --implicit-deps-unchanged, but is also more accurate. Instead of specifying this on the command line each run, you can set this option inside your SConstruct or SConscript file using "SetOption('implicit_cache', 1)".
• CPPPATH: normally you tell Scons about include directories by setting the CPPPATH construction variable, which causes SCons to search those directories when doing implicit dependency scans and also includes those directories in the compile command line. If you have header files that never or rarely change (e.g. system headers, or C run-time headers), then you can exclude them from CPPPATH and include them in the CCFLAGS construction variable instead, which causes SCons to ignore those include directories when scanning for implicit dependencies. Carefully tuning the include directories in this way can usually result in a dramatic speed increase with very little loss of accuracy.
• Avoid RCS and SCCS scans by using env.SourceCode(".", None) - this is especially interesting if you are using lots of c or c++ headers in your program and that your file system is remote (nfs, samba).
• When using "BuildDir", use it with "duplicate" set to 0: "BuildDir( dir1, dir2, duplicate=0". This will cause scons to invoke Builders using the path names of source files in src_dir and the path names of derived files within build_dir. However, this may cause build problems if source files are generated during the build, if any invoked tools are hard-coded to put derived files in the same directory as the source files.
• On a multi-processor machine it may be beneficial to run multiple jobs at once - use the --jobs N (where N is the number of processors on your machine), or "SetOption('num_jobs', N)" inside your SConstruct or SConscript. On Windows machines, the number of processors is available in the environment variable 'NUMBER_OF_PROCESSORS'.
• If you have more than a few dozen preprocessor defines ("-DFOO1 -DFOO2") you may find from using --profile that SCons is spending a lot of time in subst() function, usually just adding the -D string to the defines over and over again. This can really slow down builds where nothing has changed. With 100+ defines I saw a do-nothing build time drop from 35s to 20s using the idea described in "Caching the CPPDEFINES" elsewhere on this page.

Another trick to making things faster is to avoid relinking programs when a shared library has been modified but not rebuilt. See SharedLibrarySignatureOverride

Answer 2

scons md5-sums files to figure out they've changed, so it pretty much md5sums all your files.

You can tell it to only use timestamps to decide what to rebuild and not have to MD5sum all the files every time , much like 'make' does, which should speed things up. It might be more fragile. e.g. if a file has changed within 1 second of the last time it was built, scons won't notice that. Use

env.Decider('timestamp-newer')

There's also MD5-timestamp, which will check the timestamp first, then compare the content using Md5 if it's actually changed if the timestamp is newer.

env.Decider('MD5-timestamp')

Another easy option to speed things up is to run parallel building using the -j parameter.

scons -j 2

On my 2-core box, -j 3 usually gives the biggest speedup.

Some output on what scons is doing can be done with the --debug argument to calling scons, see the manpage for the various options.

Answer 3

Did a bit of trial and error to figure out why SCons is slow, some findings so far (exact results would of course vary depending on the structure and complexity of the SCons script):

• CacheDir() has no noticeable negative impact.
• Decider() has only very minor impact, not worth bothering with.
• Using variant_dir/VariantDir() increases the build time by around 10%.
• Reading the SConstruct file itself takes around 10% of the complete scons call.
• The by far biggest impact seems to be the library dependencies, having Gtkmm in the project doubled the build time for me.

Possible solutions:

• Don't do complete rebuilds, but only rebuild the directory/module you are working on (scons -u instead of scons -D).
• Make pieces of the SConscript optional, so its only rebuild when manually called.

• Use the compiler flag -isystem for library includes instead of -I, this change alone brought down the build time from 10.5sec to 6sec for me, it can be easily accomplished with a little sed call:

  env.ParseConfig('pkg-config --cflags --libs gtkmm-2.4 | sed "s/-I/-isystem/g"')

  Not exactly sure why this works, I assume it cuts down the dependencies that gcc outputs and thus in turn the dependencies that scons tracks.

Using CacheDir() and scons -j N is of course also highly recommend, but only speeds up the actually building, not the evaluation of the SCons script itself.

Answer 4

Moving 3rd party includes out of CPPPATH and into CCFLAGS made a huge difference. For our project with 12 external include directories (including boost and python), a do nothing compile went from 30s down to 3s - a 10x speedup.

Answer 5

When SCons creates first Environment it makes a series of lookups to see what tools are available. You can avoid unnecessary checks and speedup SCons by explicitly choosing tools in DefaultEnvironment before creating your first env.

DefaultEnvironment(tools=[])

Answer 6

Add to your SConscript something like

if 'explain' in GetOption("debug"):
    Progress('Evaluating $TARGET\n')

and run with --debug=explain. You will see what SCons spends time evaluating

Answer 7

scons --profile + snakeviz

This combination showed me exactly what the bottlneck was.

--profile outputs a binary file in cProfile format, which is present in the stdlib.

snakeviz is then an awesome visualization tool to quickly view that file in a GUI:

scons --profile f.prof
pip install -u snakeviz
snakeviz f.prof

The output looked like this:

and you can hover each box to see the full path to the file that contains the function.

Question in a more general Python context: Is there any simple way to benchmark python script?

--debug + ts -s

This didn't solve my specific problem, but it can often give you some ideas:

• enable all --debug flags, or all that seem interesting
• pipe it throuth ts -s, which shows when each line appears on stdout: How to monitor for how much time each line of stdout was the last output line in Bash for benchmarking?

time scons --debug=count,duplicate,explain,findlibs,includes,memoizer,memory,objects,prepare,presub,stacktrace,time |
  ts -s | tee f

Sample output excerpt that showed where I had a huge time gap between 2 and 10 seconds, which is where I tried to focus:

00:00:02 SConscript:/data/gem5/master3/build/ARM/sim/power/SConscript  took 1.556 ms                                       
00:00:02 dup: relinking variant 'build/ARM/sim/probe/SConscript' from 'src/sim/probe/SConscript'                              
00:00:02 Building build/ARM/sim/probe/SConscript with action:                                                                                                                
00:00:02   UnlinkFunc(target, source, env)                                                                                      
00:00:02 Building build/ARM/sim/probe/SConscript with action:                                                                  
00:00:02   LinkFunc(target, source, env)                                                                                                            
00:00:02 SConscript:/data/gem5/master3/build/ARM/sim/probe/SConscript  took 0.401 ms       
00:00:10 SConscript:/data/gem5/master3/build/ARM/tests/opt/SConscript  took 98.225 ms                                               
00:00:10 SConscript:/data/gem5/master3/build/ARM/SConscript  took 8885.387 ms            
00:00:10 SConscript:/data/gem5/master3/SConstruct  took 9409.641 ms                         
00:00:10 scons: done reading SConscript files.                                                                                    
00:00:10 scons: Building targets ...

Tested in scons 3.0.1, Ubuntu 18.04.

See also

• Windows subset: SCons super slow startup in windows