limiting file output to only 2 lines

Question

I ping a series of addresses and append the latency results to a file (each address has a separate file). I'm trying to limit the file to only contain the last 2 entries.

$outpath=/opt/blah/file.txt
resp_str="0.42"
echo $resp_str >> $outpath

tail -2 $outpath > $outpath

Without tail, the file continues to grow with the new data (simply .42 for this example). But when I call tail, it writes out an empty file. If I redirect the tail output to a file of a different name, then I get the expected result. Can I not write out to a file as I read it? Is there a simple solution?

Here's the complete script:

OUT_PATH=/opt/blah/sites/
TEST_FILE=/opt/blah/test.txt

while IFS=, read -r ip name; do
    if [[ "$ip" != \#* ]]; then
        RESP_STR=$( ping -c 1 -q $ip | grep rtt| awk '{print $4}' | awk -F/ '{ print $2; }')
        echo $RESP_STR >> "$OUT_PATH""test/"$name".txt"
    fi
done << $TEST_FILE

what do you mean *'file continues to grow'*? what (command) is writing to the file? — markp-fuso, May 03 '22 at 19:43
Echo is writing to the file. This script would be called every 5 minutes. I've just simplified the script for example purposes. resp_str is actually a parsed response from ping. — Phaelax z, May 03 '22 at 19:46
but if the script is called every 5 minutes, isn't the file going to continue to grow? or is the intent to insure the file only ever contains a max of 2 lines (ie, the 2 latest `ping` results) — markp-fuso, May 03 '22 at 19:47
That's why I'm trying to use tail, to capture the last two entries and overwrite the file so that it doesn't grow. — Phaelax z, May 03 '22 at 19:48
@pjh possibly. If I'm understanding it correctly, I'd have to write to a temp file first then back. — Phaelax z, May 03 '22 at 20:05
Also see [Bash Pitfalls #13 (cat file | sed s/foo/bar/ > file)](https://mywiki.wooledge.org/BashPitfalls#cat_file_.7C_sed_s.2Ffoo.2Fbar.2F_.3E_file). — pjh, May 03 '22 at 20:15

score 1 · Accepted Answer · answered May 03 '22 at 21:59

1

tail -2 $outpath > $outpath

> truncates the file before tail starts reading it.

You need to buffer the output of tail before writing it back to that file. Use sponge to achieve this:

tail -2 $outpath | sponge $outpath

answered May 03 '22 at 21:59

Piotr Henryk Dabrowski

702
5
13

1

For those reading this in the future, sponge is part of the moreutils package (which I had to install). Very useful command! – Phaelax z May 06 '22 at 18:24

score 0 · Answer 2 · answered May 03 '22 at 19:47

0

You can use pipe | to send the output of one commande to another, in this case tail.
We can then append out to a file using >>. If we use > we overwrite the file each time and all previous content is lost.
This example writes the 2 last files in the directory to log.txt each time it is run.

ls  | tail -2 >> log.txt

answered May 03 '22 at 19:47

I'm not following the purpose of using ls. – Phaelax z May 03 '22 at 19:55
1

I'm just using **ls** as an example of a command that will produce more than 2 lines of output to demonstrate the principle of piping to another command and then writing to file. You can put any other command in place of **ls**. – May 03 '22 at 19:58

markp-fuso · Answer 3 · 2022-05-03T20:35:14.537

Assumptions:

need to code for parallel, concurrent processes (use of temp files will require each process to have a uniquely named temp file)
go ahead and code to support a high volume of operations (ie, reduce overhead of creating/destroying temp files)

One idea using mktemp to create a temporary file ... we'll wrap this in a function for easier use:

keep2 () {

    # Usage: keep2 filename "new line of text"

    [[ ! -f "${tmpfile}" ]] &&
    tmpfile=$(mktemp)

    tail -1 "$1" > "${tmpfile}"
    { cat "${tmpfile}"; echo "$2"; } > "$1"
}

NOTES:

the hardcoded -1 (tail -1) could be parameterized or reference a user-defined env variable
OP can change the order of the input parameters as desired

Taking for a test drive;

> logfile

for ((i=1;i<=5;i++))
do
    echo "######### $i"
    keep2 logfile "$i"
    cat logfile
done

This generates:

######### 1
1
######### 2
1
2
######### 3
2
3
######### 4
3
4
######### 5
4
5

In OP's code the following line:

echo $RESP_STR >> "$OUT_PATH""test/"$name".txt"

would be replaced with:

keep2 "$OUT_PATH""test/"$name".txt" "${RESP_STR}"

limiting file output to only 2 lines

3 Answers3