Use a Glob() in VariantDir() einvironment to find files recursively in Python?

Question

How can one make recursive Glob() in a VariantDir() environment in Python?

The answer from the question <Use a Glob() to find files recursively in Python?> will not work, because you need use Glob() to get a list of files that is aware of VariantDir() environment.

So you need something like:

import fnmatch
import os

matches = []
for root, dirnames, filenames in os.walk('src'):
  for filename in fnmatch.filter(filenames, '*.c'):
    matches.append(os.path.join(root, filename))

matches = Glob(matches)

Will this work?

Answer 1

I believe the accepted answer only works if your source directory is '.'.

Here is a working example where the source directory is src and the variant directory is build.

File layout

.
├── include
│   └── sum.h
├── SConstruct
└── src
    ├── dir
    │   └── sum.cpp
    └── main.cpp

Files

sum.h

#ifndef _SUM_H_
#define _SUM_H_

double sum(const double x, const double y);

#endif

main.cpp

#include <iostream>
#include <sum.h>

using namespace std;

int main() {

    cout << "Sum of 1 and 2 = " << sum(1., 2.) << endl;
    return 0;
}

sum.cpp

#include <sum.h>

double sum(const double x, const double y) {
    return x + y;
}

SConstruct

import os

def variantglob(env, pattern, ondisk=True, source=True, strings=False,
                recursive=False):
    matches = []
    for root, dirs, filenames in os.walk(env['SOURCE_DIR']):
        cwd = Dir(os.path.join(env['VARIANT_DIR'],
                               os.path.relpath(root, env['SOURCE_DIR'])))
        matches.extend(cwd.glob(pattern, ondisk, source, strings))
    return matches

# Create Build Environment
env = Environment()

# Customize Environment
env.Replace(VARIANT_DIR='build',
            SOURCE_DIR='src')
env.Append(CPPPATH=['include'])

# Setup Variant Directory
VariantDir(variant_dir=env['VARIANT_DIR'],
           src_dir=env['SOURCE_DIR'], duplicate=0)

# Build the executable
exe = env.Program(os.path.join(env['VARIANT_DIR'], 'example'),
                  variantglob(env, '*.cpp', recursive=True))

# Install the executable
Install('bin', exe)

Build

Just execute scons at the top level directory. This will create a build directory and drop all your temporaries there (variant directory), and it will then install the result of the build into the bin folder.

Run

Execute bin/example to see it work.

Note

This example was tested on linux.

Why It Works

When building with variant directories you have to specify the path to the source as though it were already sitting in the variant directory, but those directories may not exist yet. This glob function walks the source tree to construct the paths that will be in the variant directory, and then globs against those paths.

Answer 2

Your approach would work with a minor tweak as follows:

import fnmatch
import os

def RecursiveGlob(pathname)
    matches = []
    for root, dirnames, filenames in os.walk(pathname):
        for filename in fnmatch.filter(filenames, '*.c'):
            matches.append(File(os.path.join(root, filename)))

    return matches

Notice that I converted it to a File(), since the SCons Glob() function returns Nodes if the "strings" parameter is false.

To be able to handle the VariantDir, etc and to better integrate the functionality with the existing SCons Glob() functionality, you could actually incorporate a call to the existing Glob() function, like this:

# Notice the signature is similar to the SCons Glob() signature,
# look at scons-2.1.0/engine/SCons/Node/FS.py line 1403
def RecursiveGlob(pattern, ondisk=True, source=True, strings=False):
    matches = []
    # Instead of using os.getcwd() consider passing-in a path
    for root, dirnames, filenames in os.walk(os.getcwd()):
        cwd = Dir(root)
        # Glob() returns a list, so using extend() instead of append()
        # The cwd param isnt documented, (look at the code) but its 
        # how you tell SCons what directory to look in.
        matches.extend(Glob(pattern, ondisk, source, strings, cwd))

    return matches

You could take it one step further and do the following:

def MyGlob(pattern, ondisk=True, source=True, strings=False, recursive=False):
    if not recursive:
        return Glob(pattern, ondisk, source, strings)

    matches = []
    # Instead of using os.getcwd() consider passing-in a path
    for root, dirnames, filenames in os.walk(os.getcwd()):
        cwd = Dir(root)
        # Glob() returns a list, so using extend() instead of append()
        # The cwd param isnt documented, (look at the code) but its 
        # how you tell SCons what directory to look in.
        matches.extend(Glob(pattern, ondisk, source, strings, cwd))

    return matches