bash "map" equivalent: run command on each file

Question

I often have a command that processes one file, and I want to run it on every file in a directory. Is there any built-in way to do this?

For example, say I have a program data which outputs an important number about a file:

./data foo
137
./data bar
42

I want to run it on every file in the directory in some manner like this:

map data `ls *`
ls * | map data

to yield output like this:

foo: 137
bar: 42

Answer 1

If you are just trying to execute your data program on a bunch of files, the easiest/least complicated way is to use -exec in find.

Say you wanted to execute data on all txt files in the current directory (and subdirectories). This is all you'd need:

find . -name "*.txt" -exec data {} \;

If you wanted to restrict it to the current directory, you could do this:

find . -maxdepth 1 -name "*.txt" -exec data {} \;

There are lots of options with find.

Answer 2

If you just want to run a command on every file you can do this:

for i in *; do data "$i"; done

If you also wish to display the filename that it is currently working on then you could use this:

for i in *; do echo -n "$i: "; data "$i"; done

Answer 3

It looks like you want xargs:

find . --maxdepth 1 | xargs -d'\n' data

To print each command first, it gets a little more complex:

find . --maxdepth 1 | xargs -d'\n' -I {} bash -c "echo {}; data {}"

Answer 4

You should avoid parsing ls:

find . -maxdepth 1 | while read -r file; do do_something_with "$file"; done

or

while read -r file; do do_something_with "$file"; done < <(find . -maxdepth 1)

The latter doesn't create a subshell out of the while loop.

Answer 5

The common methods are:

ls * | while read file; do data "$file"; done

for file in *; do data "$file"; done

The second can run into problems if you have whitespace in filenames; in that case you'd probably want to make sure it runs in a subshell, and set IFS:

( IFS=$'\n'; for file in *; do data "$file"; done )

You can easily wrap the first one up in a script:

#!/bin/bash
# map.bash

while read file; do
    "$1" "$file"
done

which can be executed as you requested - just be careful never to accidentally execute anything dumb with it. The benefit of using a looping construct is that you can easily place multiple commands inside it as part of a one-liner, unlike xargs where you'll have to place them in an executable script for it to run.

Of course, you can also just use the utility xargs:

find -maxdepth 0 * | xargs -n 1 data

Note that you should make sure indicators are turned off (ls --indicator-style=none) if you normally use them, or the @ appended to symlinks will turn them into nonexistent filenames.

Answer 6

GNU Parallel specializes in making these kind of mappings:

parallel data ::: *

It will run one job on each CPU core in parallel.

GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to.

If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:

GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:

Installation

If GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README

Learn more

See more examples: http://www.gnu.org/software/parallel/man.html

Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html

Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel

Answer 7

Since you specifically asked about this in terms of "map", I thought I'd share this function I have in my personal shell library:

# map_lines: evaluate a command for each line of input
map_lines()
{
        while read line ; do
                $1 $line
        done
}

I use this in the manner that you for a solution:

$ ls | map_lines ./data

I named it map_lines instead of map as I assumed some day I may implement a map_args where you would use it like this:

$ map_args ./data *

That function would look like this:

map_args()
{
    cmd="$1" ; shift
    for arg ; do
        $cmd "$arg"
    done
}

Answer 8

Try this:

for i in *; do echo ${i}: `data $i`; done

Answer 9

You can create a shell script like so:

#!/bin/bash
cd /path/to/your/dir
for file in `dir -d *` ; do
  ./data "$file"
done

That loops through every file in /path/to/your/dir and runs your "data" script on it. Be sure to chmod the above script so that it is executable.

Answer 10

You could also use PRLL.

Answer 11

ls doesn't handle blanks, linefeeds and other funky stuff in filenames and should be avoided where possible.

find is only useful if you like to dive into subdirs, or if you want to make usage from the other options (mtime, size, you name it).

But many commands handle multiple files themself, so don't need a for-loop:

for d in * ; do du -s $d; done

but

du -s *
md5sum e* 
identify *jpg
grep bash ../*.sh

Answer 12

I have just written this script specifically to address the same need.

http://gist.github.com/kindaro/4ba601d19f09331750bd

It uses find to build a set of files to transpose, which allows for finer selection of files to map from but allows a window for harder mistakes as well.

I designed two modes of operation: the first mode runs a command with "source file" and "target file" arguments, while the second mode supplies source file contents to a command as stdin and writes its stdout into a target file.

We may further consider adding support for parallel execution and maybe limiting the set of custom find arguments to a few most necessary ones. I am not really sure if that's the right things to do.