1

Closest I could find to this is Is it possible to mimic process substitution on msys /mingw (with bash 3.x) - but I cannot really get it to work in my case; so let me demonstrate.


I'm using:

$ bash --version
GNU bash, version 5.1.8(1)-release (x86_64-pc-msys)

I usually have a specific use for process substitution and diff (as in the linked question) - usually I want to call head or tail to compare parts of files. As a starting point we might have:

cat > /tmp/tst <<EOF
line 1
line 2
line 3
EOF

So, I might want to call something like this - which I think is great, because it is a (readable for me) one-liner, which allows pre-processing files for diff:

$ diff <(head -n 3 /tmp/tst) <(head -n 2 /tmp/tst)
3d2
< line 3

This works here - however, for network storage with large files it might fail ... additionally, note that process substitution usually replaces the <(...) with a filename:

$ echo <(head -n 3 /tmp/tst)
/dev/fd/63

... however, some programs might receive the "Windows" path, as in via cygpath -w:

$ echo $(cygpath -w <(head -n 3 /tmp/tst))
/proc/694/fd/pipe:[249108113544]

... like for instance meld (MINGW64):

meld <(head -n 3 /tmp/tst) <(head -n 2 /tmp/tst)

The above command starts meld - and we would have expected /dev/fd/* filenames in the call to meld - however, when meld opens, there are the following messages in the two panes:

There was a problem opening the file “\proc\701\fd\63”.        | There was a problem opening the file “\proc\701\fd\63”.
Error opening file \proc\701\fd\62: No such file or directory  | Error opening file \proc\701\fd\62: No such file or directory

... so, meld ultimately received the /proc/* filenames, as if passed via cygpath -w - and failed with those filenames.


So, I thought - how about using file descriptors directly? As an example:

## note, fd 3 is typically used in MSYS2:

$ ls /dev/fd/
3@  0@  1@  2@

## assign fd 10 (open for reading) to the process substitution of `head`:
# ( note that `exec 10><(head -n 2 /tmp/tst)` causes "bash: /dev/fd/63: Permission denied" )
# https://stackoverflow.com/questions/7082001/how-do-file-descriptors-work
# https://unix.stackexchange.com/questions/208758/how-to-set-pipe-of-process-after-process-substitution-in-bash-so-that-process-co

$ exec 10< <(head -n 2 /tmp/tst)

## check descriptors

$ ls /dev/fd/
3@  0@  1@  10@  2@

## check once more /dev/fd/ for fd 10:

$ ls /dev/fd/10
/dev/fd/10@

$ ls -la /dev/fd/10
lrwxrwxrwx 1 user None 0 Jan 14 13:59 /dev/fd/10 -> 'pipe:[515396091548]'|

## cat/output fd 10 via /dev/fd/ filename does NOT work!:

$ cat /dev/fd/10
cat: /dev/fd/10: No such file or directory

## cat/output fd 10 via redirection - replacing stdin of cat - works:

$ cat <&10
line 1
line 2

## cat second time - nothing is printed, since process already finished
$ cat <&10
$

## close fd 10 (close for reading, since it was open for reading)
# http://mywiki.wooledge.org/BashFAQ/085
# https://unix.stackexchange.com/questions/13724/file-descriptors-shell-scripting

$ exec 10<&-

## check descriptors again:
$ ls /dev/fd/
3@  0@  1@  2@

The above snippet shows the core of the problem - how come, that /dev/fd/10 exists when used with ls -la - but when used with cat, it seemingly does not exist ("No such file or directory")?

Otherwise, I would have hoped something like this - using /dev/fd/* filenames, when redirecting process output to file descriptor - would have worked for a one-liner (where I can ultimately preprocess the input files for diff):

$ exec 10< <(head -n 3 /tmp/tst) && exec 11< <(head -n 2 /tmp/tst) && ls /dev/fd/ && ls -la /dev/fd/10 /dev/fd/11 && exe
c 10<&- && exec 11<&- && ls /dev/fd/
3@  0@  1@  10@  11@  2@
lrwxrwxrwx 1 user None 0 Jan 14 14:02 /dev/fd/10 -> 'pipe:[528280993436]'|
lrwxrwxrwx 1 user None 0 Jan 14 14:02 /dev/fd/11 -> 'pipe:[532575960732]'|
3@  0@  1@  2@

... however, as seen with the /dev/fd/ filename problem above, this invocation fails with diff (and fails also if you replace diff with cat below):

$ exec 10< <(head -n 3 /tmp/tst) && exec 11< <(head -n 2 /tmp/tst) && ls /dev/fd/ && diff /dev/fd/10 /dev/fd/11 && exec
10<&- && exec 11<&- && ls /dev/fd/
3@  0@  1@  10@  11@  2@
diff: /dev/fd/10: No such file or directory
diff: /dev/fd/11: No such file or directory

So, ultimately, my question is - is there a way to use a file descriptor instead of a filename in a bash call of a command? (because if so, I could use file descriptors instead of process substitution in bash to run one-liner operations on truncated files, where process substitution - while otherwise working - fails on some programs on MSYS2).

sdbbs
  • 4,270
  • 5
  • 32
  • 87
  • 1
    Depends on the operating system and the program you're calling. On real Linux systems, `/proc` is a real filesystem which `/dev/fd` symlinks into; on msys it's emulated by the C library, so it's a lot less "real" -- things calling Windows syscalls can't see it. If you're not on a system where `/proc/self/fd` is a reliable kernel-backed abstraction, I wouldn't count on being able to make this work everywhere you want it. – Charles Duffy Jan 14 '22 at 13:37
  • 1
    To be clear, this isn't really something bash can control -- executing programs is just passing an array of C strings to it; how the program interprets that an array of strings is up to it. – Charles Duffy Jan 14 '22 at 13:39
  • 1
    Did you try something like `meld "$(echo <(head -n 3 /tmp/tst) | cygpath -w)" "$(echo <(head -n 2 /tmp/tst) | cygpath -w)"`? If it's the windows version of `meld` it would need Windows style paths. There's also the Linux version of `meld`. – dan Jan 14 '22 at 15:21
  • 1
    Maybe you could named pipes instead? – dan Jan 14 '22 at 15:27
  • Thanks all - great points emphasized! @dan : this is the MINGW64 version of `meld` - that is, Python code is the same, Gtk libraries are "Linux", but MINGW64 compiled (I think; I know there is difference between MSYS2 compilation, where binaries require msys2.dll, and MINGW*, where they don't and hook to native .dll. About "named pipes" I'm not sure, I don't know them - can you give a brief example of them? – sdbbs Jan 14 '22 at 16:04
  • 1
    To answer your *first* question: the file descriptor numbers are not global, they are specific to your particular process. So when you use `exec` to set up file descriptor 10, that file descriptor is only visible to your *Bash* process. If you open up `/dev/fd/10` in another process, it won't exist or it will be something different, unless you specifically told Bash to pipe something into the new process's fd 10 using Bash syntax. – David Grayson Jan 14 '22 at 19:01
  • 1
    And if you're using a MINGW progam like meld, that is a native Windows program that probably won't know about the file descriptors set up by the POSIX-emulation part of MSYS2. You'll probably just need to use temporary files on the disk. – David Grayson Jan 14 '22 at 19:21

0 Answers0