Bash- why can't a file be overwritten without the use of a temp file?

Question

The standard procedure for overwriting a file is usually the following:

awk '{print $2*3}' file > tmp file
tmpFile > file

However, sometimes this poses to be a bit of a hassle because then one must remove the temp file after it is no longer being used.

So, why is it not possible to do this in the following way (without the need of a temp file) :

awk '{print $2*3}' file > file

The reason I ask is because I know that it is possible to append to a file as so:

awk '{print $2*3}' file >> file

So if appending a file, using >> as shown above, works fine, why can't one overwrite a file in the same way. Why are the two commands so different?

Moreover, does there exist a way of bypassing the need for a temp file (perhaps in a fashion similar to the 2nd excerpt), or is the first excerpt the only way?

NOTE: the awk command is irrelevant, it can be replaced by any other command

See [*this*](http://unix.stackexchange.com/questions/11067/is-there-a-way-to-modify-a-file-in-place) — Nir Alfasi, Jan 18 '15 at 18:09
possible duplicate of [After pipes, how to overwrite the original file? why does it work some some platform and doesnt in others?](http://stackoverflow.com/questions/10586623/after-pipes-how-to-overwrite-the-original-file-why-does-it-work-some-some-plat) — tripleee, Jan 18 '15 at 19:35
@tripleee you're right it is a duplicate, but I didn't know that when I formulated the question. Thanks for the down vote anyways. I flagged it as well. — buydadip, Jan 18 '15 at 19:51

score 1 · Accepted Answer · edited Jan 18 '15 at 19:28

1

Using a temp file is a good idea because you can never be sure if the entire file will be read into memory. If you try to write it before it was read, then you might get a different result than you might have expected.

When using append, the command always goes through the entire file before adding new content, so there never remains a part of the file to be read.

edited Jan 18 '15 at 19:28

buydadip

8,890
22
79
154

answered Jan 18 '15 at 18:18

oxym

86
1
3

score 1 · Answer 2 · answered Jan 18 '15 at 18:19

1

Probably not a great idea (trying to read & write to same file), but if you insist on doing it, you could use the <> operator.

gawk '{print $2*3}' -- <> file

answered Jan 18 '15 at 18:19

user590028

11,364
3
40
57

score 0 · Answer 3 · answered Jul 19 '22 at 20:24

There's a tool for everything. You can use sponge.

awk '{print $2*3}' file | sponge file

You can get it from the moreutils package. The man page reads:

NAME
       sponge - soak up standard input and write to a file

SYNOPSIS
       sed '...' file | grep '...' | sponge [-a] file

DESCRIPTION
       sponge reads standard input and writes it out to the specified file.
       Unlike a shell redirect, sponge soaks up all its input before writing the
       output file. This allows constructing pipelines that read from and write
       to the same file.

       sponge preserves the permissions of the output file if it already exists.

       When possible, sponge creates or updates the output file atomically by
       renaming a temp file into place. (This cannot be done if TMPDIR is not in
       the same filesystem.)

       If the output file is a special file or symlink, the data will be written
       to it, non-atomically.

       If no file is specified, sponge outputs to stdout.

OPTIONS
       -a
           Replace the file with a new file that contains the file's original
           content, with the standard input appended to it. This is done
           atomically when possible.

AUTHOR
       Colin Watson and Tollef Fog Heen

score 0 · Answer 4 · answered Jul 20 '22 at 01:28

if u happen to be on a mac, you can emulate a copy & paste operation to do in-place edits indirectly without a temp file :

awk '{ . . . }' file | LC_ALL=C pbcopy ; LC_ALL=C pbpaste > file

dunno what the equivalent commands are for linux or other platforms. avoid this if your file is over 500 MB in size

you can also use this for perl or python etc since "pasteboard copy" is simply reading in contents via /dev/stdin.

this is only a convenience shortcut and doesn't guarantee atomic ops whatsoever.

Bash- why can't a file be overwritten without the use of a temp file?

4 Answers4