Recursive wildcards in GNU make?

Question

It's been a while since I've used make, so bear with me...

I've got a directory, flac, containing .FLAC files. I've got a corresponding directory, mp3 containing MP3 files. If a FLAC file is newer than the corresponding MP3 file (or the corresponding MP3 file doesn't exist), then I want to run a bunch of commands to convert the FLAC file to an MP3 file, and copy the tags across.

The kicker: I need to search the flac directory recursively, and create corresponding subdirectories in the mp3 directory. The directories and files can have spaces in the names, and are named in UTF-8.

And I want to use make to drive this.

Any reason for selecting make for this purpose? I'd have thought writing a bash script would be simpler — , Mar 20 '10 at 13:37
(...or I could write it in Ruby or Python). I'd like to have a play with make beyond the basics, and this is a 'project' I have open right now. — Roger Lipscombe, Mar 20 '10 at 13:40
@Neil, make's concept as pattern-based file system transformation is the best way to approach the original problem. Perhaps implementations of this approach have its limitations, but `make` is closer to implementing it than bare `bash`. — P Shved, Mar 20 '10 at 13:55
@Pavel Well, a `sh` script that walks through the list of flac files (`find | while read flacname`), makes a `mp3name` from that, runs "mkdir -p" on the `dirname "$mp3name"`, and then, `if [ "$flacfile" -nt "$mp3file"]` converts `"$flacname"` into `"$mp3name"` is not really magic. The only feature you are actually losing compared to a `make` based solution is the possibility to run `N` file conversions processes in parallel with `make -jN`. — ndim, Mar 20 '10 at 15:45
Admittedly, `make`'s declarative approach is nicer than any imperative language could ever offer. — ndim, Mar 20 '10 at 15:46
@ndim That's the first time I have ever heard make's syntax be described as "nice" :-) — , Mar 20 '10 at 16:05
Using `make` and having spaces in file names are contradictory requirements. Use a tool appropriate for the problem domain. — Jens, Aug 15 '13 at 17:45
Related: https://stackoverflow.com/questions/3774568/makefile-issue-smart-way-to-scan-directory-tree-for-c-files — Gabriel Devillers, Nov 30 '18 at 20:29

score 139 · Accepted Answer · edited Jun 19 '12 at 12:58

139

I would try something along these lines

FLAC_FILES = $(shell find flac/ -type f -name '*.flac')
MP3_FILES = $(patsubst flac/%.flac, mp3/%.mp3, $(FLAC_FILES))

.PHONY: all
all: $(MP3_FILES)

mp3/%.mp3: flac/%.flac
    @mkdir -p "$(@D)"
    @echo convert "$<" to "$@"

A couple of quick notes for make beginners:

The @ in front of the commands prevents make from printing the command before actually running it.
$(@D) is the directory part of the target file name ($@)
Make sure that the lines with shell commands in them start with a tab, not with spaces.

Even if this should handle all UTF-8 characters and stuff, it will fail at spaces in file or directory names, as make uses spaces to separate stuff in the makefiles and I am not aware of a way to work around that. So that leaves you with just a shell script, I am afraid :-/

edited Jun 19 '12 at 12:58

zrajm

1,361
1
12
21

answered Mar 20 '10 at 13:35

ndim

35,870
12
47
57

This is where I was going...fast fingers for the win. Though it looks like you may be able to do something clever with `vpath`. Must study that one of these days. – dmckee --- ex-moderator kitten Mar 20 '10 at 13:41
2

Doesn't appear to work when the directories have spaces in the names. – Roger Lipscombe Mar 20 '10 at 13:45
1

Didn't realize that I'd have to shell out to `find` to get the names recursively... – Roger Lipscombe Mar 20 '10 at 13:47
Oh. Spaces. Well, make will not work with spaces. Make syntax uses spaces for its own purposes. – ndim Mar 20 '10 at 13:47
@Roger: No it doesn't. There is a Grouch Marx skit involving a doctor... But I suppose that the file naming is not in your control. – dmckee --- ex-moderator kitten Mar 20 '10 at 13:49
What, even in the filenames? Ick. – Roger Lipscombe Mar 20 '10 at 13:49
Fair enough; I'll investigate other options. Marking this as the answer anyway. – Roger Lipscombe Mar 20 '10 at 13:50
See the follow up question about rake, instead: http://stackoverflow.com/questions/2483418/recursive-wildcards-in-rake – Roger Lipscombe Mar 20 '10 at 14:40
Does make not use lazy evaluation on dependency lists? I put something like this into my Makefile and it still runs the find command even if I tell make to build a target that doesn't need it (and yes, I am using '=' instead of ':='). – SystemParadox Feb 20 '12 at 10:54
Perfect. I needed a variant on it that turns one subtree's coffee into another subtree's javascript: https://gist.github.com/johan/5490763 – ecmanaut Apr 30 '13 at 18:27
Is there a way to parallelize and/or asynchronize the conversion recipe? It's rather slow when I run it in sequence but when I send all the file names to the convert command directly it's many fold faster. – paulkon Nov 19 '13 at 15:52
1

@PaulKonova: Run `make -jN`. For `N` use the number of conversions which `make` should run in parallel. Caution: Running `make -j` without an `N` will start all conversion processes at once in parallel which might be equivalent to a fork bomb. – ndim Nov 19 '13 at 20:40
Yeah, that works. Also, in this particular case, is there a way to remove old directories and files from the MP3 directory which are no longer in the FLAC directory? To have the MP3 directory maintain parity with the FLAC directory? – paulkon Nov 19 '13 at 21:46
@PaulKonova Not with ´make`. – ndim Nov 20 '13 at 00:56
`FLAC_FILES = $(shell find flac -type f -name '*.flac')` remove the `/` after flac folder if you don't want something like that (double slashes): `flac//file.flac` – mems Jun 18 '14 at 09:10
What is the line `.PHONY: all` for? – Adrian Torrie Feb 12 '18 at 00:26
1

@Adrian: The `.PHONY: all` line tells make that the recipe for the `all` target is to be executed even if there is a file called `all` newer than all the `$(MP3_FILES)`. – ndim Apr 26 '18 at 11:27
thank you, been trying to do exactly this for an hour already. glad I stumbled upon this. – Mel Mar 30 '19 at 11:35

score 79 · Answer 2 · edited Oct 10 '19 at 16:37

79

You can define your own recursive wildcard function like this:

rwildcard=$(foreach d,$(wildcard $(1:=/*)),$(call rwildcard,$d,$2) $(filter $(subst *,%,$2),$d))

The first parameter ($1) is a list of directories, and the second ($2) is a list of patterns you want to match.

Examples:

To find all the C files in the current directory:

$(call rwildcard,.,*.c)

To find all the .c and .h files in src:

$(call rwildcard,src,*.c *.h)

This function is based on the implementation from this article, with a few improvements.

edited Oct 10 '19 at 16:37

HolyBlackCat

78,603
9
131
207

answered Aug 15 '13 at 17:37

This doesn't seem to work for me. I've copied the exact function and it still won't look recursively. – Jeroen Nov 05 '13 at 19:38
5

I am using GNU Make 3.81, and it seems to work for me. It won't work if any of the filenames have spaces in them, though. Note that the filenames it returns have paths relative to the current directory, even if you are only listing files in a subdirectory. – Nov 13 '13 at 06:53
3

This is truly an example, that `make` is a Turing Tar Pit (see here: http://yosefk.com/blog/fun-at-the-turing-tar-pit.html). It is not even that hard, but one has to read this: https://www.gnu.org/software/make/manual/html_node/Call-Function.html and then "understand recurrence". YOU had to write this recursively, in the verbatim sense; it's not the everyday understanding of "automatically include stuff from subdirs". It's actual RECURRENCE. But remember - "To understand recurrence, you have to understand recurrence". – Tomasz Gandor Aug 06 '14 at 22:13
1

@TomaszGandor You don't have to understand recurrence. You have to understand recursion and in order to do that you must first understand recursion. – user1129682 May 07 '19 at 15:19
My bad, I fell for a linguistic false-friend. Comments can't be edited after such a long time, but I hope everybody got the point. And they also understand recursion. – Tomasz Gandor May 09 '19 at 08:42
This is more portable than calling a shell function. Also I was able to use it to get a list of directories `$(sort $(dir $(call rwildcard,src,*)))` (where `src` is my top level dir) – Rodney May 15 '21 at 12:46
Despite the tar pit, this is the right answer because it is portable and does not depend on shell commands. – Kenn Sebesta Feb 26 '22 at 23:11
Interestingly, this command seems to fail for search strings which have an `_` in them. For instance, `$(call rwildcard,src,hw*.h)` works, but `$(call rwildcard,src,hw_*.h)` does not. This is not the case for a normal wildcard call, e.g. `$(wildcard src/hw_*.h)`. – Kenn Sebesta Feb 26 '22 at 23:23

kenorb · Answer 3 · 2017-11-30T13:02:34.100

6

If you're using Bash 4.x, you can use a new globbing option, for example:

SHELL:=/bin/bash -O globstar
list:
  @echo Flac: $(shell ls flac/**/*.flac)
  @echo MP3: $(shell ls mp3/**/*.mp3)

This kind of recursive wildcard can find all the files of your interest (.flac, .mp3 or whatever). O

edited Nov 30 '17 at 13:02

answered May 10 '15 at 18:01

kenorb

155,785
88
678
743

3

To me, even just $(wildcard flac/**/*.flac) seems to work. OS X, Gnu Make 3.81 – akauppi May 13 '15 at 11:10
2

I tried $(wildcard ./**/*.py) and it behaved the same as $(wildcard ./*/*.py). I don't think make actually supports **, and it just doesn't fail when you use two *s next to each other. – lahwran Nov 22 '16 at 21:06
@lahwran It should when you invoking commands via Bash shell and you've enabled _globstar_ option. Maybe you're not using GNU make or something else. You may also try [this syntax](http://superuser.com/a/619851/87805) instead. Check the comments for some suggestions. Otherwise it's a thing for the new question. – kenorb Nov 23 '16 at 00:20
@kenorb no no, I didn't even try your thing because I wanted to avoid shell invocation for this particular thing. I was using akauppi's suggested thing. The thing I went with looked like larskholte's answer, though I got it from somewhere else because the comments here said this one was subtly broken. shrug :) – lahwran Nov 23 '16 at 01:12
1

@lahwran In this case `**` won't work, because the extended globbing is a bash/zsh thing. – kenorb Nov 23 '16 at 11:06
Just want to chime in that `$(wildcard flac/**/*.flac)` does not work for me on OS X Catalina with Gnu Make 4.2.1. Which is a bummer :/ – Sơn Trần-Nguyễn Feb 07 '20 at 14:17
@SơnTrần-Nguyễn Make sure to enable it with `shopt -s globstar` and try on shell first. Check option by `shopt | grep globstar`. – kenorb Feb 07 '20 at 16:46
@kenorb Catalina switched the shell from bash to zsh. I did test `ls css/**/*.css` and it worked. – Sơn Trần-Nguyễn Feb 07 '20 at 17:10
Bash switched to GNU license with version 4.0 so Apple is stuck on 3.2. You have to upgrade it yourself or switch to a different shell (switch to bash with a newer version? upgrading feels dangerous). – grofte Apr 04 '23 at 12:05

score 2 · Answer 4 · edited Jul 19 '15 at 12:06

My solution is based on the one above, uses sed instead of patsubst to mangle the output of find AND escape the spaces.

Going from flac/ to ogg/

OGGS = $(shell find flac -type f -name "*.flac" | sed 's/ /\\ /g;s/flac\//ogg\//;s/\.flac/\.ogg/' )

Caveats:

Still barfs if there are semi-colons in the filename, but they're pretty rare.
The $(@D) trick won't work (outputs gibberish), but oggenc creates directories for you!

Brecht Machiels · Answer 5 · 2017-06-06T07:54:09.610

Here's a Python script I quickly hacked together to solve the original problem: keep a compressed copy of a music library. The script will convert .m4a files (assumed to be ALAC) to AAC format, unless the AAC file already exists and is newer than the ALAC file. MP3 files in the library will be linked, since they are already compressed.

Just beware that aborting the script (ctrl-c) will leave behind a half-converted file.

I originally also wanted to write a Makefile to handle this, but since it cannot handle spaces in filenames (see the accepted answer) and because writing a bash script is guaranteed to put in me in a world of pain, Python it is. It's fairly straightforward and short, and thus should be easy to tweak to your needs.

from __future__ import print_function


import glob
import os
import subprocess


UNCOMPRESSED_DIR = 'Music'
COMPRESSED = 'compressed_'

UNCOMPRESSED_EXTS = ('m4a', )   # files to convert to lossy format
LINK_EXTS = ('mp3', )           # files to link instead of convert


for root, dirs, files in os.walk(UNCOMPRESSED_DIR):
    out_root = COMPRESSED + root
    if not os.path.exists(out_root):
        os.mkdir(out_root)
    for file in files:
        file_path = os.path.join(root, file)
        file_root, ext = os.path.splitext(file_path)
        if ext[1:] in LINK_EXTS:
            if not os.path.exists(COMPRESSED + file_path):
                print('Linking {}'.format(file_path))
                link_source = os.path.relpath(file_path, out_root)
                os.symlink(link_source, COMPRESSED + file_path)
            continue
        if ext[1:] not in UNCOMPRESSED_EXTS:
            print('Skipping {}'.format(file_path))
            continue
        out_file_path = COMPRESSED + file_path
        if (os.path.exists(out_file_path)
            and os.path.getctime(out_file_path) > os.path.getctime(file_path)):
            print('Up to date: {}'.format(file_path))
            continue
        print('Converting {}'.format(file_path))
        subprocess.call(['ffmpeg', '-y', '-i', file_path,
                         '-c:a', 'libfdk_aac', '-vbr', '4',
                         out_file_path])

Of course, this can be enhanced to perform the encoding in parallel. That is left as an exercise to the reader ;-)

score 2 · Answer 6 · answered Dec 03 '11 at 20:09

FWIW, I've used something like this in a Makefile:

RECURSIVE_MANIFEST = `find . -type f -print`

The example above will search from the current directory ('.') for all "plain files" ('-type f') and set the RECURSIVE_MANIFEST make variable to every file it finds. You can then use pattern substitutions to reduce this list, or alternatively, supply more arguments into find to narrow what it returns. See the man page for find.

score 0 · Answer 7 · answered Aug 25 '23 at 11:42

To find files recursively without resorting to external dependencies like find, you can use functions. Then use the result as in the other answer to convert the files.

rwildcard=$(wildcard $1) $(foreach d,$1,$(call rwildcard,$(addsuffix /$(notdir $d),$(wildcard $(dir $d)*))))

FLAC_FILES = $(call rwildcard,flac/*.flac)
MP3_FILES = $(patsubst flac/%.flac, mp3/%.mp3, $(FLAC_FILES))

.PHONY: all
all: $(MP3_FILES)

mp3/%.mp3: flac/%.flac
        @mkdir -p "$(@D)"
        @echo convert "$<" to "$@"

See https://github.com/markpiffer/gmtt#call-wildcard-reclist-of-globs for a beefed up version of recursive wildcards — Vroomfondel, Aug 28 '23 at 08:35

Recursive wildcards in GNU make?

7 Answers7

Examples:

Linked