22

I tried to use the read/write file descriptor in bash so that I could delete the file that the file descriptor referred to afterward, as such:

F=$(mktemp)
exec 3<> "$F"
rm -f "$F"

echo "Hello world" >&3
cat <&3

but the cat command gives no output. I can achieve what I want if I use separate file descriptors for reading and writing:

F=$(mktemp)
exec 3> "$F"
exec 4< "$F"
rm -f "$F"

echo "Hello world" >&3
cat <&4

which prints Hello world.

I suspected that bash doesn't automatically seek to the start of the file descriptor when you switch from writing to reading it, and the following combination of bash and python code confirms this:

fdrw.sh

exec 3<> tmp
rm tmp

echo "Hello world" >&3
exec python fdrw.py

fdrw.py

import os  

f = os.fdopen(3)
print f.tell()
print f.read()

which gives:

$ bash fdrw.sh
12

$ # This is the prompt reappearing

Is there a way to achieve what I want just using bash?

telotortium
  • 3,383
  • 2
  • 23
  • 25
  • 1
    why would you want to delete the file before reading/writing? – unhammer Mar 21 '11 at 10:20
  • 6
    In Unix, when you remove a file, the file isn't actually deleted until all open file descriptors to it are closed. Thus, deleting a temporary file right after opening is common practice, since it guarantees that no other process can maliciously alter the file and that the file is closed after your process closes the file or exits. – telotortium Mar 21 '11 at 20:16
  • 1
    Why don't you like your own method of having separate read and write descriptors? That seems like the simplest way. – Kelvin May 14 '12 at 16:37

9 Answers9

12

I found a way to do it in bash, but it's relying on an obscure feature of exec < /dev/stdin which actually can rewind the file descriptor of stdin according to http://linux-ip.net/misc/madlug/shell-tips/tip-1.txt:

F=$(mktemp)
exec 3<> "$F"
rm -f "$F"

echo "Hello world" >&3
{ exec < /dev/stdin; cat; } <&3

The write descriptor isn't affected by that so you can still append output to descriptor 3 before the cat.

Sadly I only got this working under Linux not under MacOS (BSD), even with the newest bash version. So it doesn't seem very portable.

David Ongaro
  • 3,568
  • 1
  • 24
  • 36
10

If you ever do happen to want to seek on bash file descriptors, you can use a subprocess, since it inherits the file descriptors of the parent process. Here is an example C program to do this.

seekfd.c

#define _FILE_OFFSET_BITS 64
#include <string.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <unistd.h>

int main(int argc, char* argv[])
{
    /* Arguments: fd [offset [whence]]
     * where
     * fd: file descriptor to seek
     * offset: number of bytes from position specified in whence
     * whence: one of
     *  SEEK_SET (==0): from start of file
     *  SEEK_CUR (==1): from current position
     *  SEEK_END (==2): from end of file
     */
    int fd;
    long long scan_offset = 0;
    off_t offset = 0;
    int whence = SEEK_SET;
    int errsv; int rv;
    if (argc == 1) {
        fprintf(stderr, "usage: seekfd fd [offset [whence]]\n");
        exit(1);
    }
    if (argc >= 2) {
        if (sscanf(argv[1], "%d", &fd) == EOF) {
            errsv = errno;
            fprintf(stderr, "%s: %s\n", argv[0], strerror(errsv));
            exit(1);
        }
    }
    if (argc >= 3) {
        rv = sscanf(argv[2], "%lld", &scan_offset);
        if (rv == EOF) {
            errsv = errno;
            fprintf(stderr, "%s: %s\n", argv[0], strerror(errsv));
            exit(1);
        }
        offset = (off_t) scan_offset;
    }
    if (argc >= 4) {
        if (sscanf(argv[3], "%d", &whence) == EOF) {
            errsv = errno;
            fprintf(stderr, "%s: %s\n", argv[0], strerror(errsv));
            exit(1);
        }
    }

    if (lseek(fd, offset, whence) == (off_t) -1) {
        errsv = errno;
        fprintf(stderr, "%s: %s\n", argv[0], strerror(errsv));
        exit(2);
    }

    return 0;
}
telotortium
  • 3,383
  • 2
  • 23
  • 25
  • 1
    I'm certian there would be aperl oneliner that will do the same thing and be more portible – anthony Jan 06 '21 at 07:13
  • @anthony I'm not sure about the "more portable". It's a lot easier to cross-compile a C program written to the standard without extensions than to cross-compile the Perl interpreter (which, back in the day, was once inside my job description -- at an embedded systems shop where we targeted hardware that wasn't large enough to compile perl natively; porting the Perl interpreter was not a fun one). – Charles Duffy Aug 16 '22 at 12:27
  • I suppose it depends on your point of view. Portible to me is not needing to re-compile a program for multiple systems, just using it directly, multiple Linuxes, Macs, Solaris, etc etc etc. But I can see you point of view too.. A link to a perl version is given in my answer (really simple). this version also lets you use 'tell', or 'truncate' and can use names rather than numbers. – anthony Oct 06 '22 at 10:43
5

When you open a file descriptor in bash like that, it becomes accessible as a file in /dev/fd/. On that you can do cat and it'll read from the start, or append (echo "something" >> /dev/fd/3), and it'll add it to the end. At least on my system it behaves this way. (On the other hand, I can't seem to be able to get "cat <&3" to work, even if I don't do any writing to the descriptor).

Petr Skocik
  • 58,047
  • 6
  • 95
  • 142
5

No. bash does not have any concept of "seeking" with its redirection. It reads/writes (mostly) from beginning to end in one long stream.

Ignacio Vazquez-Abrams
  • 776,304
  • 153
  • 1,341
  • 1,358
  • 2
    Basically, then, the only reason for read/write descriptors in bash is to pass them to an exec'ed process? – telotortium Oct 01 '10 at 10:59
  • 1
    In order to provide more channels than just stdin, stdout, and sterr, yes. – Ignacio Vazquez-Abrams Oct 01 '10 at 11:37
  • It is also commonly used with network connections... `exec {w}<>/dev/tcp/www.google.com/80` And you can use it to write to the MIDDLE of a file. You open it read/write, read N chars, then write to it. Search for "Read and Write" in https://antofthy.gitlab.io/info/shell/file_handles.txt for an example. – anthony Jan 06 '21 at 07:06
5

Try changing the sequence of commands:

F=$(mktemp tmp.XXXXXX)
exec 3<> "$F"
echo "Hello world" > "$F"
rm -f "$F"

#echo "Hello world" >&3
cat <&3
yabt
  • 51
  • 1
  • 7
    @Dennis this solution actually works. The cat isn't reading from the apparently deleted file. It's reading from the descriptor that's still open. You can still access a file's contents with that descriptor, even though the last (hard) link to it has been removed. – Kelvin May 14 '12 at 16:44
  • 1
    Problem with this solution is that the `echo` can take a long time, which means the temporary file stays on the filesystem for a long time. If this is acceptable then you can simply use filename, not fd 3, for redirection. – Cyker Dec 10 '19 at 08:44
3
#!/bin/bash
F=$(mktemp tmp.XXXXXX)
exec 3<> $F
rm $F

echo "Hello world" >&3
cat /dev/fd/3

As suggested in other answer, cat will rewind the file descriptor for you before reading from it since it thinks it's just a regular file.

Community
  • 1
  • 1
sanmai
  • 29,083
  • 12
  • 64
  • 76
1

To 'rewind' the file descriptor, you can simply use /proc/self/fd/3

Test script :

#!/bin/bash

# Fill data
FILE=test
date +%FT%T >$FILE

# Open the file descriptor and delete the file
exec 5<>$FILE
rm -rf $FILE

# Check state of the file
# should return an error as the file has been deleted
file $FILE

# Check that you still can do multiple reads or additions
for i in {0..5}; do
    echo ----- $i -----

    echo . >>/proc/self/fd/5
    cat /proc/self/fd/5

    echo
    sleep 1
done

Try to kill -9 the script while it is running, you will see that contrary to what happens with the trap method, the file is actually deleted.

Hexdump
  • 159
  • 1
  • 7
0

Expansion on the answer by @sanmai...

And confirmation of what is going on...

#/bin/bash
F=$(mktemp tmp.XXXXXX)
exec 3<>$F     # open the temporary file for read and write
rm $F          # delete file, though it remains on file system

echo "Hello world!" >&3    # Add a line to a file
cat /dev/fd/3              # Read the whole file
echo "Bye" >>/dev/fd/3     # Append another line
cat /dev/fd/3              # Read the whole file
echo "Goodbye" >&3         # Overwrite second line
cat /dev/fd/3              # Read the whole file

cat <&3                    # Try to Rewind (no output)
echo "Cruel World!" >&3    # Still adds a line on end
cat /dev/fd/3              # Read the whole file

shell_seek 3 6 0           # seek fd 3 to position 6
echo -n "Earth" >&3        # Overwrite 'World'
shell_seek 3               # rewind fd 3
cat <&3                    # Read the whole file put 3 at end

Note that the echo Goodbye overwrites the second lineas the file descriptor &3 had not changed by the cat!

So I tried using cat <&3 which did not output anything, probably as the file descriptor was at the end of the file. To see it if it rewinds the descriptor it was given. It does not.

The last part is to use the 'C' program that was provided, compiled and named shell_seek and yes it seems it works as the first 'World' was replaced by 'Earth', the rewind (seek to start) worked allowing the last cat to again read the whole file. It would put the fd at the end of the file again!

Doing it using perl instead of C was not that hard either. For example perl -e 'open(FD,">&3"); seek(FD,0,0);' will rewind file descriptor 3 back to the start of the file.

I have now made a perl version of shell_seek so I don't have to re-compile it all the time for different systems. Not only that but the script can also 'tell' you the current file descriptor offset, and also 'truncate' the file that file descriptor points too. Both operations are commonly used when using seek, so it seemed a good idea to include those functions. You can download the script from... https://antofthy.gitlab.io/software/#shell_seek

anthony
  • 7,696
  • 1
  • 17
  • 11
  • `cat /dev/fd/3` doesn't rewind the descriptor in macOS (BSD?), so it's also not portable (I even tested with GNU `cat` and it still doesn't work). Curiously, it works with `tail -r` even for `<&3`, so one could do something like `tail -r <&3 | tail -r` but it looks ugly and there is no `-r` option for GNU `tail`. Alternatively, one could do `tail -100000 <&3` but choosing an arbitrary number here also looks ugly (or one has to know the max number of expected lines). `tail +0 <&3` doesn't work for some reason. Also, with GNU `tail` under macOS it doesn't work, so be sure to use `/usr/bin/tail`. – David Ongaro Nov 15 '22 at 19:40
  • Thanks for the update about MacOS. Better to use the shell_seek C program, or expanded perl script provided to more precisely control the file descriptor. – anthony Nov 18 '22 at 01:08
0

En utilisant un tube nommé
mkfifo tubeA exec 3 <>tubeA rm tubeA
echo "Hello world" >&3 read x <&3 echo $x
echo "Hello world" >&3 cat <&3
^C

dmabboux
  • 1
  • 1