Continuously paste to a file in a loop without overwriting

Question

I would like to paste an output file from a loop side to side with each file generated from the loop without overwriting the file.

If it was just append it would be fine with >> but struggling to find a way where I can keep adding to a file so columns side by side

cat a.txt >> final.txt

The files I have are in the format

File 1     file 2
123        108
176        193
123        145

I would like to paste these together

paste -d ' ' file1.txt file2.txt > final.txt

I would then like to continue adding to the file final with iterations through the loop

paste -d ' ' file3.txt final.txt >>final.txt

The output I would like then would have 3 columns- i.e file3.txt is not appended to the bottom thus increasing rows but added to the side like in a normal paste so the column no increases with each pase

123 108 125    
176 193 478
123 145 645

Any help would be much appreciated

I case it's not obvious, `paste -d ' ' file3.txt file1.txt file2.txt >final.txt` does what you want in a single step. — tripleee, Aug 12 '21 at 10:26
Yes but this loop runs 1000s of times so was hoping to avoid keeping file3.txt file1.txt and so on till the end... Currently I paste the XXthousand at the end using for f in *_${chr}_3.txt; do cat final_${chr}.txt | paste -d ' ' - $f >temp; cp temp final_${chr}.txt; done; but Im taking a lot of unnecessary space if i could just paste within the loop and keep adding to the file — dcp1234, Aug 13 '21 at 11:46
That sounds horribly inefficient. Get the file names in the order you want them and `paste` them in one go. (You may have to break it up into multiple passes still if you have more files than you can fit into `ARG_MAX` before hitting "Argument list too long".) — tripleee, Aug 13 '21 at 11:49
how many files? what's the total size of these files? do all files have the same number of lines? — markp-fuso, Sep 05 '21 at 13:31

score 0 · Answer 1 · answered Aug 12 '21 at 10:21

0

In regular case you can append to a file like this:

cat a.txt &>> final.txt

But if you would like to get columns, you should use paste, e.g. to create .csv file:

paste -d , final.csv a.csv

.txt files are not the usual format for holding columns, so you should consider the .csv format.

answered Aug 12 '21 at 10:21

Attis

573
1
7
19

Thank you but I dont think I was clear and have amended the question- cat will append a directly underneath the last row in final.txt - I dont want to increase the rows I want to directly paste the values side by side continuously so row length stays the same in the final.txt but the number of columns increases as a file is pasted through each iteration. – dcp1234 Aug 12 '21 at 10:37

score 0 · Answer 2 · answered Sep 05 '21 at 12:55

It's not at all clear why you'd want to do this instead of creating an output file once at the end of all processing but assuming it's useful:

seq 2 > file
for i in {1..3}; do
    seq 2 > tmp1 &&
    paste file tmp1 > tmp2 &&
    mv tmp2 file
    echo "-----"
    cat file
done
-----
1       1
2       2
-----
1       1       1
2       2       2
-----
1       1       1       1
2       2       2       2

score 0 · Answer 3 · answered Sep 05 '21 at 12:57

truggling to find a way where I can keep adding to a file so columns side by side

You have to use a temporary file. (Or you could use sponge from moreutils.)

paste -d ' ' file3.txt final.txt >> final.txt.temp
mv final.txt.temp final.txt

currently I paste the XXthousand at the end using for f in *${chr}3.txt; do cat final${chr}.txt | paste -d ' ' - $f >temp; cp temp final${chr}.txt; done;

Just run it once.

paste -d ' ' *_${chr}_3.txt > final_${chr}.txt

tripleee · Answer 4 · 2021-09-05T13:32:27.300

The absolutely simplest and most efficient solution is to simply run paste once, with all your input files as arguments.

Based on what you said in a comment, you seem to be looking simply for

paste -d ' ' *_"${chr}_3.txt" >"final_${chr}.txt"

If getting the files in a different order is important, try to organize the argument to paste first, perhaps by using an array. Here's with the files in reverse order:

files=( *_"${chr}_3.txt" )
reversed=( )
for((i=${#files[@]}; i>0; i--)); do
    reversed+=("${files[i]}")
done
paste -d ' ' "${reversed[@]}" >"final_${chr}.txt"

Arrays are a Bash-only feature (though also portable to ksh and more or less to Zsh); the earlier solution above is portable to any Bourne shell. This for loop syntax is also a Bash extension.

If you have a lot of files or your file names are very long, you may bump into "argument list too long" errors. The fix for that is to use xargs, though it can be a bit tricky to adapt to this scenario. Maybe also look at GNU parallel which has some features to simplify this sort of scenario. As an alternative, here's a simple recursive function which avoids any explicit temporary files.

ppaste () {
    case $# in
     1|2) paste -d ' ' "$@";;
     *) local first=$1
        shift
        paste -d ' ' "$first" <(ppaste "$@");;
    esac
}

The process substitution <(...) is again a Bash feature which isn't portable to POSIX sh.

Demo: https://ideone.com/FEwKYo

Continuously paste to a file in a loop without overwriting

4 Answers4