How to write the output of a bash command to a csv file?

Question

I currently follow memory usage of my machine using the following loop in bash :

echo "      date     time $(free -m | grep total | sed -E 's/^    (.*)/\1/g')" | tee -a log20201113.txt
while true; do
    echo "$(date '+%Y-%m-%d %H:%M:%S') $(free -m | grep Mem: | sed 's/Mem://g')" | tee -a log20201113.txt
    sleep 1
done

Which give me a txt file with something like that (modified values) :

      date     time          total       used       free     shared    buffers     cached
2020-11-13 22:55:56           8333        627       5705          0       1638        685
2020-11-13 22:55:57           8333        677       5656          0       1638        685
2020-11-13 22:55:58           8333        725       5607          0       1638        685
2020-11-13 22:55:59           8333        773       5560          0       1638        685

I am trying to output that to a csv file instead. Putting a csv extension doesn't seems to work because it doesn't include separators. How would I include separators and put that in a csv file instead ?

score 1 · Answer 1 · answered Nov 14 '20 at 16:45

try piping in sed:

echo "      date     time $(free -m | grep total | sed -E 's/^    (.*)/\1/g')" | tee -a log20201113.txt
while true; do
    echo "$(date '+%Y-%m-%d %H:%M:%S') $(free -m | grep Mem: | sed 's/Mem://g')" | tee -a log20201113.txt
    sleep 1
done | sed "s/\s\+/,/g"

the sed will replace spaces by commas, then you hace you csv.

from the text file directly you may do:

sed -i "s/\s\+/,/g" yourfile.txt, it will replace all spaced by comma.

then you get

2020-11-14,17:42:42,7748,2190,729,601,4828,4490
2020-11-14,17:42:43,7748,2189,733,598,4825,4494

score 1 · Accepted Answer · answered Nov 14 '20 at 16:50

piping to awk '{$1=$1}1' OFS="," before tee should be enough to convert to csv. e.g:

echo "      date     time $(free -m | grep total | sed -E 's/^    (.*)/\1/g')" | awk '{$1=$1}1' OFS="," | tee -a log20201113.csv
while true; do
    echo "$(date '+%Y-%m-%d %H:%M:%S') $(free -m | grep Mem: | sed 's/Mem://g')" | awk '{$1=$1}1' OFS="," | tee -a log20201113.csv
    sleep 1
done

produces:

$ head -n 5 log20201113.csv
date,time,total,used,free,shared,buff/cache,available
2020-11-14,17:30:04,3638,1663,765,177,1208,1561
2020-11-14,17:30:05,3638,1663,765,177,1208,1561
2020-11-14,17:30:06,3638,1662,766,177,1208,1562
2020-11-14,17:30:07,3638,1662,767,177,1208,1562

when awk assigns to a field ($1=$1) it will

causes $0 to be reconstructed by concatenating the $i's separated by OFS

where $0 is the input line, $i are the fields and OFS is , in our case. the terminating 1 is just an "always match" patter that always gets executed with the default action (print).

but this will also print to stdout in csv format. if this is a problem and you would like to keep nicely formatted output on the terminal and csv in the file, you can use this "trick":

echo "      date     time $(free -m | grep total | sed -E 's/^    (.*)/\1/g')" | tee /dev/tty | awk '{$1=$1}1' OFS="," >> log20201113.csv
while true; do
    echo "$(date '+%Y-%m-%d %H:%M:%S') $(free -m | grep Mem: | sed 's/Mem://g')" | tee /dev/tty | awk '{$1=$1}1' OFS="," >> log20201113.csv
    sleep 1
done

score 0 · Answer 3 · answered Nov 14 '20 at 17:10

Here's one in GNU awk, that reads the mem info from /proc/meminfo:

$ gawk '
@load "time"                                  # sleep
BEGIN {
    OFS=","                                   # field separator
    printf "%s%s%s%s%s%s%s",                  # header
        "date time" OFS,
        "total" OFS,
        "used" OFS,
        "free" OFS,
        "shared" OFS,
        "buffers" OFS,
        "cached" ORS
    while(1) {                                # keep looping
        while((getline < "/proc/meminfo")>0)
            a[$1]=$2
        printf "%s%s%s%s%s%s%s",
            strftime("%F%T") OFS,
            a["MemTotal:"] OFS,
            a["MemTotal:"]-a["MemFree:"]-a["Buffers:"]-a["Cached:"]-a["SReclaimable"] OFS,
            a["MemFree:"] OFS,
            a["ShmemHugePages:"] OFS,
            a["Buffers:"] OFS,
            a["Buffers:"]+a["SReclaimable:"] ORS
        sleep(1)
    }
}'

Output:

date time,total,used,free,shared,buffers,cached
2020-11-1419:12:42,16135228,6586304,6713024,0,213652,362516
2020-11-1419:12:43,16135228,6586304,6713024,0,213652,362516
...

How to write the output of a bash command to a csv file?

3 Answers3