Scons. Go recursive with Glob

Question

I using scons for a few days and confused a bit. Why there is no built-in tools for building sources recursively starting from given root? Let me explain: I have such source disposition:

src
    Core
       folder1
       folder2
           subfolder2_1
    Std
       folder1

..and so on. This tree could be rather deeper.

Now I build this with such construction:

sources = Glob('./builds/Std/*/*.cpp')
sources = sources + Glob('./builds/Std/*.cpp')
sources = sources + Glob('./builds/Std/*/*/*.cpp')
sources = sources + Glob('./builds/Std/*/*/*/*.cpp')

and this looks not so perfect as at can be. Of cause, I can write some python code, but is there more suitable ways of doing this?

use [os.walk()](http://docs.python.org/library/os.html) maybe? — elmo, May 15 '12 at 11:05
@elmo, yes I find some snippets to solve the problem. But there is still the question why this is not the out-of-box feature? I mean if I doing something wrong? — Alex Povar, May 15 '12 at 11:44
Probably not wrong, but hierarchical builds tend to be more popular for some reason — Kos, May 15 '12 at 11:44
I gonna to use hierarchical builds one level upper to build main project from subprojects. But in this case I have just a tree of sources in java-like manner. — Alex Povar, May 15 '12 at 12:06
Presumably, your sub-directories would build something your current directory build depends on. Have a hierarchy of SConscripts, each of which builds from *.cpp and Return the result to form the dependency above. — Evgen, Jun 04 '21 at 06:46

score 10 · Answer 1 · answered Jun 15 '12 at 17:50

10

As Torsten already said, there is no "internal" recursive Glob() in SCons. You need to write something yourself. My solution is:

import fnmatch
import os

matches = []
for root, dirnames, filenames in os.walk('src'):
  for filename in fnmatch.filter(filenames, '*.c'):
    matches.append(Glob(os.path.join(root, filename)[len(root)+1:]))

I want to stress that you need Glob() here (not glob.glob() from python) especially when you use VariantDir(). Also when you use VariantDir() don't forget to convert absolute paths to relative (in the example I achieve this using [len(root)+1:]).

answered Jun 15 '12 at 17:50

Igor Chubin

61,765
13
122
144

2

This almost did it for me, although I had to change my append argument to `Glob(os.path.join(root, filename))`. Your code was giving me blank arrays. Thanks though. – Olical Oct 27 '12 at 09:18
You don't need to use Glob once you have walked the tree. :) – wallabra May 08 '19 at 19:07

score 5 · Accepted Answer · edited Mar 10 '15 at 14:57

Sure. You need to write python wrappers to walking through dirs. You can find many recipes on stackoverflow. Here is my simple function which returns list of subdirs in present dir (and ignore hide dirs starting with '.' - dot)

def getSubdirs(abs_path_dir) :  
    lst = [ name for name in os.listdir(abs_path_dir) if os.path.isdir(os.path.join(abs_path_dir, name)) and name[0] != '.' ]
    lst.sort()
    return lst

For example, i've dir modules what containts foo, bar, ice.

corePath = 'abs/path/to/modules'
modules = getSubdirs(corePath)
# modules = [bar, foo, ice]
for module in modules :
  sources += Glob(os.path.join(corePath, module, '*.cpp'))

You can improve getSubdirs function adding recurse and walking deeper to subdirs.

score 3 · Answer 3 · edited Mar 21 '13 at 23:12

3

The Glob() SCons function doesnt have the ability to go recursive.

It would be much more efficient if you change your Python code to use the list.extend() function, like this:

sources = Glob('./builds/Std/*/*.cpp')
sources.extend(Glob('./builds/Std/*.cpp'))
sources.extend(Glob('./builds/Std/*/*/*.cpp'))
sources.extend(Glob('./builds/Std/*/*/*/*.cpp'))

Instead of trying to go recursive like you are, its quite common to have a SConscript script in each subdirectory and in the root SConstruct call each of them with the SConscript() function. This is called a SCons hierarchical build.

edited Mar 21 '13 at 23:12

Patrick O'Leary

223
2
6

answered May 15 '12 at 11:43

Brady

10,207
2
20
59

1

Should I use SConscript for every "package" of source? They are all built in same way without any per-package difference. – Alex Povar May 15 '12 at 12:09
1

@AlexPovar, This approach is useful for allowing different compilation flags, defines, include paths, etc per directory. Additionally, you can "ship" separate directories with their individual build scripts. It may be excessive to have a SConscript per source directory, but you should have more than one global SConstruct. – Brady May 15 '12 at 13:01

Piotr Siupa · Answer 4 · 2022-08-21T16:50:47.347

Here is my version of recursive Glob:

from SCons.Environment import Base as BaseEnvironment

def __RGlob(self, root_path, pattern, ondisk=True, source=False, strings=False, exclude=None):
    result_nodes = []
    paths = [root_path]
    while paths:
        path = paths.pop()
        all_nodes = self.Glob(f'{path}/*', ondisk=ondisk, source=source, exclude=exclude)
        paths.extend(entry for entry in all_nodes if entry.isdir() or (entry.srcnode() and entry.srcnode().isdir())) # `srcnode()` must be used because `isdir()` doesn't work for entries in variant dirs which haven't been copied yet.
        result_nodes.extend(self.Glob(f'{path}/{pattern}', ondisk=ondisk, source=source, strings=strings, exclude=exclude))
    return sorted(result_nodes)
BaseEnvironment.RGlob = __RGlob

It tries imitate the standard Glob as closely as possible. The biggest difference is that beside the pattern is takes a root path as another argument. The pattern is then applied to this root path and every sub-directory in it.

This code adds function RGlob it to the base environment, which means you will be able to call it on every environment that is created after that. The best place to paste this is probably the file site_scons/site_init.py.

score 1 · Answer 5 · answered Oct 23 '19 at 17:30

1

I use this:

srcdir = './'
sources = [s for s in glob2.glob(srcdir + '**/*.cpp') if "/." not in s]

answered Oct 23 '19 at 17:30

Nir

69
5

While this is a simpler solution, not using SCons' Glob() will miss handling variant dirs, and also miss any files not yet built which SCons knows about. – bdbaddog Oct 23 '19 at 21:04

score 0 · Answer 6 · answered Aug 07 '23 at 23:43

I use this solution:

env = Environment()
build_dir = 'build'
src_dir = 'src'
env.VariantDir(build_dir, src_dir, duplicate=0)
env = SConscript("SConstruct")

sources = Glob("build/*.cpp")
for root, dirnames, filenames in os.walk(src_dir):
    root = root[len(src_dir)+1:]
    for dir in dirnames:
        p = os.path.join(build_dir, root, dir, '*.cpp')
        sources.extend(Glob(p))

Scons. Go recursive with Glob

6 Answers6

Linked