0
FILES=../path/*
for FILE in $FILES;
do
./processInput.py -f <(./processInput.py -f $FILE) > $FILE
done

Above is the gist of my code. I am redirecting system output from one process into a temporary file which is used as input to the same script but in a different process. When I redirect that output to some random file, things work out fine and the file contains the correct output. However, when I try to redirect back to $FILE, it errors out as if the outer process has an issue. Is there some order of operations that's messing things up here?

I could output to some random file and then remove the file when i'm done with it, but i'd like to avoid that unnecessary step if possible.

Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
Skorpius
  • 2,135
  • 2
  • 26
  • 31
  • You cannot read/write back to the same file at the same time – Inian Jun 13 '18 at 17:05
  • That's basically an alternate way to write `./processInput.py -f "$FILE" | ./processInput.py -f - >"$FILE"`, which doesn't work for the same reasons (even if your program recognizes the conventional meaning of `-` referring to stdin/stdout as positionally appropriate). – Charles Duffy Jun 13 '18 at 17:05
  • All parts of a pipeline run at the same time so `> "$FILE"` blanks the file when the pipeline is first started, so `-f "$FILE"` is reading from an empty file. – Charles Duffy Jun 13 '18 at 17:06
  • ...which is to say, this problem is in no way whatsoever specific to process substitution (the `<(...)` syntax used). – Charles Duffy Jun 13 '18 at 17:07
  • ...now, what you *could* do is `./processInput.py -f <(./processInput.py -f "$FILE") >"$FILE.$$" && mv -- "$FILE.$$" "$FILE"`; notably, writing to a different output and then renaming is the same fix suggested in the general (non-process-substitution) case as well. – Charles Duffy Jun 13 '18 at 17:08
  • I assume this is outputting to some temporary file then moving that temporary file to replace the file I want? – Skorpius Jun 13 '18 at 17:10
  • Exactly, yes. That way you don't blank the original file at the start of operation, preventing it from being used for input. Safer to use a real temporary file created by `mktemp`, of course (since that uses `O_CREAT|O_EXCL` to prevent symlink attacks), but the `$$` hack fits into a comment easier. :) – Charles Duffy Jun 13 '18 at 17:11
  • Edit: Saw you answered my question. Thanks! Would you recommend I do it that way or use mktemp? For the sake of readability. Also is there any resources on the "$FILE.$$" thing you're doing? I couldn't find it on a google search. Thanks for the help! – Skorpius Jun 13 '18 at 17:14
  • `$$` is the shell's PID, so it's safer to use as a temporary file name than just `$FILE.tmp`, albeit not as safe as `tempfile=$(mktemp "$FILE.XXXXXX")` and then using `"$tempfile"`. The difference is in whether an attacker can figure out a name you're likely to use (PIDs are quite predictable) and create a symlink to a file you have permissions to overwrite but they can't. – Charles Duffy Jun 13 '18 at 17:15
  • ...btw, as an aside, all-caps variable names are actually not great form; see http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html, fourth paragraph -- all-caps names are used for variables with meaning to the system or shell, whereas names with at least one lowercase character are reserved for application use and guaranteed not to change a (POSIX-compliant) shell's behavior in unwanted ways. (Think about what happens if you run `for PATH in /*/; do` -- that can't happen if your variable is named `path` instead). – Charles Duffy Jun 13 '18 at 17:17

0 Answers0