47

I'm trying to write a bash script that will process a list of files whose names are stored one per line in an input file, something the likes of

find . -type f -mtime +15 > /tmp/filelist.txt
for F in $(cat /tmp/filelist.txt) ; do
  ...
done;

My problem is that filenames in filelist.txt may contain spaces, so the snipped above will expand the line

my text file.txt

to three different filenames, my, text and file.txt. How can I fix that?

agnul
  • 12,608
  • 14
  • 63
  • 85
  • 1
    You know, directory entries can also contain newlines, as well as spaces and other wacky characters. The only things a directory entry (file name) can't have are the "/" and the "" characters. (slash and null). – chris Nov 06 '09 at 17:51
  • Yep, but the files I'm working on are created by windows users on my LAN accessing a samba share, so there's a limit to filename weirdness – agnul Nov 06 '09 at 18:23
  • You can also create an array from a file cf.: https://stackoverflow.com/questions/30988586/creating-an-array-from-a-text-file-in-bash See also: http://mywiki.wooledge.org/BashFAQ/005#Loading_lines_from_a_file_or_stream – Roland Mar 05 '20 at 09:08

7 Answers7

58

Use read:

while read F  ; do
        echo $F
done </tmp/filelist.txt

Alternatively use IFS to change how the shell separates your list:

OLDIFS=$IFS
IFS="
"
for F in $(cat /tmp/filelist.txt) ; do
  echo $F
done
IFS=$OLDIFS

Alternatively (as suggested by @tangens), convert the body of your loop into a separate script, then use find's -exec option to run if for each file found directly.

Douglas Leeder
  • 52,368
  • 9
  • 94
  • 137
  • 1
    Although I voted it up, I found this script fail on paths that contain non-ASCII characters. – kakyo Nov 10 '13 at 19:11
  • Note that a newline character (line break) is expected even at the end of last line, which on Linux systems requires pressing an extra Enter to create and an empty line at the end of the file to avoid an off-by-one error. – mirekphd Nov 12 '22 at 19:32
7

You can do this without a temporary file using process substitution:

while read F
do
  ...
done < <(find . -type f -mtime +15)
Dennis Williamson
  • 346,391
  • 90
  • 374
  • 439
  • Neat! Didn't know about process substitution. – agnul Nov 06 '09 at 19:09
  • 2
    Note that the process substitution feature is a bash extension, and is not even available with bash in sh-compatibility mode. You must start your script with `#!/bin/bash` for it to work. BTW, I also recommend using `while IFS="" read -r F` to avoid possible problems with whitespace at the beginning or end of filenames, and backslashes at the end (although if the files are coming from Windows, backslashes are probably not possible). – Gordon Davisson Nov 06 '09 at 20:19
6

use while read

cat $FILE | while read line
do
echo $line
done

You can do redirect instead of cat with a pipe

DVK
  • 126,886
  • 32
  • 213
  • 327
3

You could use the -exec parameter of find and use the file names directly:

find . -type f -mtime +15 -exec <your command here> {} \;

The {} is a placeholder for the file name.

tangens
  • 39,095
  • 19
  • 120
  • 139
  • The problem is that I'm not running a single command and I would end up trying all types of weird combinations to find the right way to quote and escape the stuff after -exec – agnul Nov 06 '09 at 19:22
3

pipe your find command straight to while read loop

find . -type f -mtime +15 | while read -r line
do
   printf "do something with $line\n"
done
ghostdog74
  • 327,991
  • 56
  • 259
  • 343
0

I'm not a bash expert by any means ( I usually write my script in ruby or python to be cross-platform), but I would use a regex expration to escape spaces in each line before you process it.

For Bash Regex: http://www.linuxjournal.com/node/1006996

In a similar situation in Ruby ( processing a csv file, and cleaning up each line before using it):

File.foreach(csv_file_name) do |line| 
    clean_line = line.gsub(/( )/, '\ ') 
    #this finds the space in your file name and escapes it    
    #do more stuff here
end
konung
  • 6,908
  • 6
  • 54
  • 79
-1

I believe you can skip the temporary file entirely and just directly iterate over the results of find, i.e.:

for F in $(find . -type f -mtime +15) ; do
  ...
done;

No guarantees that my syntax is correct but I'm pretty sure the concept works.

Edit: If you really do have to process the file with a list of filenames and can't simply combine the commands as I did above, then you can change the value of the IFS variable--it stands for Internal Field Separator--to change how bash determines fields. By default it is set to whitespace, so a newline, space, or tab will begin a new field. If you set it to contain only a newline, then you can iterate over the file just as you did before.

qid
  • 1,883
  • 10
  • 15