Efficiently rereading (or otherwise reusing) content from an input file in bash

Question

I would like to apply a commands to files in a directory, target_dir, by the following code.

for t_file in $(find 'target_dir' -maxdepth 1 -type f);
do
  exec {fd}<'command_list.txt'
  while read -u "${fd}" eval_command
  do
    eval "${eval_command}"
  done
  exec {fd}>&-
done

A example of command_list.txt is

# command_list.txt
cat "${t_file}"

The program loads the command_file.txt for every files but I expects that it is more efficient if I can move the file pointer back to the first line of the file without needing to close and reopen it between iterations.

exec {fd}<'command_list.txt'

for t_file in $(find 'target_dir' -maxdepth 1 -type f);
do

  (move cursor of read to the first line of 'command_list.txt')

  while read -u "${fd}" eval_command
  do
    eval "${eval_command}"
  done

done

exec {fd}>&-

Is seeking a file pointer back to the beginning of a file without reopening it possible in bash?

`for file in $(...)` is, in and of itself, buggy -- see [DontReadLinesWithFor](http://mywiki.wooledge.org/DontReadLinesWithFor). See [UsingFind](http://mywiki.wooledge.org/UsingFind) for numerous correct alternate approaches to execute code for every `find` result. — Charles Duffy, Oct 05 '16 at 03:21
@Charles Duffy: Thank you for commenting. I will read instruction you recommended. And I added a imaginary code to explain my question. I hope it make it easy to tell my question. Thank you. — mora, Oct 05 '16 at 03:28

score 2 · Accepted Answer · edited May 23 '17 at 12:00

To answer the literal question: You can seek a file descriptor in bash only with a loadable module (plugin) to the shell adding a new builtin, or by spinning up an external program (inheriting your file descriptors) and asking it to do the seek operation (an approach given in this answer). However, the cost of spinning up an external program is larger than the cost of just closing and reopening your file, so that's not approach that really makes sense in this case.

If you want to store your command list, just do that -- store it as an array. If by "moving the cursor" you're referring to seeking within the input file, bash doesn't provide a seek primitive in the default set of builtins -- but there's no particular need for one anyhow.

# read command_list.txt only once, storing its contents in an array
readarray -t eval_commands < command_list.txt

while IFS= read -r -d '' t_file <&3; do          # iterate over filenames from find
  for eval_command in "${eval_commands[@]}"; do  # iterate over your array
    eval "$eval_command"
  done
done 3< <(find target_dir -maxdepth 1 -type f -print0)

By the way -- if you were going to pass your filename as an argument to the command rather than substituting it in, you'd want to do so as follows:

# put the filename expansion in *single quotes*
eval "$eval_command "'"$t_file"'

...or as follows:

# generate a safe version of the filename
printf -v t_file_q '%q' "$t_file"

# ...and use that safe version instead of the original
eval "$eval_command $t_file_q"

If one ran eval "$eval_command $t_file" -- not following either of these precautions -- a file in created with touch $'target_dir/hello world $(rm -rf $HOME)' would be very bad news.

Thank you for telling me how to code it. I suppose you avoid a problem of word splitting using array. But I need to time to understand it executing your code. If you don't mind, could you tell me what name is < <(...) in (done 3< <(find target_dir ...)). I would like to read its instruction but I don't know how to search it without its name. Thank you very much. — mora, Oct 05 '16 at 03:55
This is a process substitution. See http://wiki.bash-hackers.org/syntax/expansion/proc_subst, or http://mywiki.wooledge.org/ProcessSubstitution — Charles Duffy, Oct 05 '16 at 03:56

score 0 · Answer 2 · answered Oct 05 '16 at 03:14

0

find 'target_dir' -maxdepth 1 -type f -exec cat {} \;

That would cat every file in the target directory 1 folder deep.

answered Oct 05 '16 at 03:14

Stephan

531
3
16

Thank you for answering. My actual command_list.txt has some lines. And if I can dissociate scanning files and commands applying to them, it is convenient for me. Anyway, thank you for telling me the technique. – mora Oct 05 '16 at 03:22
You could always do a loop - cat commands_list.txt | while read line; do find 'target_dir' -maxdepth 1 -type f -exec "${line}" {} \; ;done – Stephan Oct 05 '16 at 03:31
@Stephan, that wouldn't work with the example given in the question where the line includes a parameter expansion intended to be `eval`'d by the shell. – Charles Duffy Oct 05 '16 at 03:36
True, but having the same variable in the command list as in the script isn't very efficient. It means you can't change the variable in the script if you want to and you're repeating yourself. Your command list should only be the command to run. In your script is where you present the variable, or file, to run against. You don't need to eval things...... `for c in \`cat command_list.txt\`; do ${c} "${t_file}";done` – Stephan Oct 05 '16 at 04:51

Efficiently rereading (or otherwise reusing) content from an input file in bash

2 Answers2