5

I'm trying to use process substitution for an input file to a program, and it isn't working. Is it because some programs don't allow process substitution for input files?

The following doesn't work:

bash -c "cat meaningless_name"
    >sequence1
    gattacagattacagattacagattacagattacagattacagattacagattaca
    >sequence2
    gattacagattacagattacagattacagattacagattacagattacagattaca
bash -c "clustalw -align -infile=<(cat meaningless_name) -outfile=output_alignment.aln -newtree=output_tree.dnd"
    (Less verbose output, finishing with:
    No sequences in file. No alignment!

But the following controls do work:

bash -c "clustalw -align -infile=meaningless_name -outfile=output_alignment.aln -newtree=output_tree.dnd"
    (Verbose output, finishing with:
    CLUSTAL-Alignment file created  [output_alignment.aln]
bash -c "cat <(cat meaningless_name) > meaningless_name2"
diff meaningless_name meaningless_name2
    (No output: the two files are the same)
bash -c "clustalw -align -infile=meaningless_name2 -outfile=output_alignment.aln -newtree=output_tree.dnd"
    (Verbose output, finishing with:
    CLUSTAL-Alignment file created  [output_alignment.aln]

Which suggest that process substitution itself works, but that the clustalw program itself doesn't like process substitution - perhaps because it creates a non-standard file, or creates files with an unusual filename.

Is it common for programs to not accept process substitution? How would I check whether this is the issue?

I'm running GNU bash version 4.0.33(1)-release (x86_64-pc-linux-gnu) on Ubuntu 9.10. Clustalw is version 2.0.10.

Andrew Grimm
  • 78,473
  • 57
  • 200
  • 338

2 Answers2

8

Process substitution creates a named pipe. You can't seek into a named pipe.

Dennis Williamson
  • 346,391
  • 90
  • 374
  • 439
4

Yes. I've noticed the same thing in other programs. For instance, it doesn't work in emacs either. It gives "File exists but can not be read". And it's definitely a special file, for me /proc/self/fd/some_number. And it doesn't work reliably in either less nor most, with default settings.

For most: 

most <(/bin/echo 'abcdef')

and shorter displays nothing. Longer values truncate the beginning. less apparently works, but only if you specify -f.

I find zsh's = much more useful in practice. It's syntactically the same, except = instead of <. But it just creates a temporary file, so support doesn't depend on the program.

EDIT:

I found zsh uses TMPPREFIX to choose the temporary filename. So even if you don't want your real /tmp to be tmpfs, you can mount one for zsh.

Matthew Flaschen
  • 278,309
  • 50
  • 514
  • 539
  • Is the temporary file a file on the hard drive, or in RAM only? I was hoping to avoid hitting the hard drive. – Andrew Grimm Nov 25 '10 at 06:22
  • For me it's `/tmp/zshXXX`. So you could use [tmpfs](http://en.wikipedia.org/wiki/Tmpfs) pretty easily. – Matthew Flaschen Nov 25 '10 at 06:25
  • 1
    The [ *zsh* process substitution documentation](http://zsh.sourceforge.net/Doc/Release/Expansion.html#SEC67) details some of the problems programs might encounter when using the `<()` form of process redirection: expecting lseek(2) to work, closing fds before processing arguments (a possible security feature), hung sub-processes if the command never reads the substituted fd. The first seems like the most likely for the case of this question. – Chris Johnsen Nov 25 '10 at 06:36
  • The computer doesn't have zsh (I'm planning on writing the files I'm using in `/dev/shm` instead), but for the benefit of other people in my situation: is there a risk that if I modify [TMPPREFIX](http://zsh.sourceforge.net/Doc/Release/Parameters.html#SEC98) it'll change where all new temporary files are kept, not just the ones created using `=(echo 'foo')`? – Andrew Grimm Nov 25 '10 at 23:25
  • @Andrew, the docs say it's used by zsh (not other programs) for all temporary files. I don't know off hand any other cases when zsh will use such files. – Matthew Flaschen Nov 26 '10 at 04:38