How to concatenate all lines from a file in Bash?

Question

I have a file csv :

data1,data2,data2
data3,data4,data5
data6,data7,data8

I want to convert it to (Contained in a variable):

variable=data1,data2,data2%0D%0Adata3,data4,data5%0D%0Adata6,data7,data8

My attempt :

data=''
cat csv | while read line
do
data="${data}%0D%0A${line}"
done
echo $data  # Fails, since data remains empty (loop emulates a sub-shell and looses data)

Please help..

score 21 · Answer 1 · answered Nov 04 '11 at 17:10

21

Simpler to just strip newlines from the file:

tr '\n' '' < yourfile.txt > concatfile.txt

answered Nov 04 '11 at 17:10

Marc B

356,200
43
426
500

1

`tr '\n' '' < csv` prints out the value, like your original script did. If you didn't need it in a variable in bash – sounds like you do – you'd have been able to forego it entirely. – paulmelnikow Nov 04 '11 at 17:43
3

Didn't "just work" for me, but led me to `tr -d '\n'` which worked great: thanks! – Tomislav Nakic-Alfirevic Jul 10 '19 at 11:05

Andrew Schulman · Accepted Answer · 2011-11-04T17:34:31.250

6

In bash,

data=$(
while read line
do
  echo -n "%0D%0A${line}"
done < csv)

In non-bash shells, you can use `...` instead of $(...). Also, echo -n, which suppresses the newline, is unfortunately not completely portable, but again this will work in bash.

edited Nov 04 '11 at 17:34

answered Nov 04 '11 at 17:13

Andrew Schulman

3,395
1
21
23

Getting extra spaces : this,is,some,data%0D%0A this,is,again,data%0D%0A third,line,of,data%0D%0A fourth,line,in,data%0D%0A – Yugal Jindle Nov 04 '11 at 17:33
Right... doh. Edited to add `echo -n`. – Andrew Schulman Nov 04 '11 at 17:34
2

FYI, `$()` is standard. Has been for years. If you want a portable `echo -n` use `printf` – sorpigal Nov 04 '11 at 17:51

score 4 · Answer 3 · answered Nov 04 '11 at 18:17

Some of these answers are incredibly complicated. How about this.

 data="$(xargs printf ',%s' < csv | cut -b 2-)"

or

 data="$(tr '\n' ',' < csv | cut -b 2-)"

Too "external utility" for you?

IFS=$'\n', read -d'\0' -a data < csv

Now you have an array! Output it however you like, perhaps with

data="$(tr ' ' , <<<"${data[@]}")"

Still too "external utility?" Well fine,

data="$(printf "${data[0]}" ; printf ',%s' "${data[@]:1:${#data}}")"

Yes, printf can be a builtin. If it isn't but your echo is and it supports -n, use echo -n instead:

data="$(echo -n "${data[0]}" ; for d in "${data[@]:1:${#data[@]}}" ; do echo -n ,"$d" ; done)"

Okay, now I admit that I am getting a bit silly. Andrew's answer is perfectly correct.

score 3 · Answer 4 · edited Jan 06 '16 at 00:30

3

I would much prefer a loop:

for line in $(cat file.txt); do echo -n $line; done

Note: This solution requires the input file to have a new line at the end of the file or it will drop the last line.

edited Jan 06 '16 at 00:30

Julien Carsique

3,915
3
22
28

answered Oct 01 '14 at 09:23

Alexandre Mélard

12,229
3
36
46

1

@Julien, you could prepend "echo "" >> file.txt" – Alexandre Mélard Jan 06 '16 at 14:29
2

It is not always possible to modify the source file. You could also use a temporary copy. Anyway, you're solution is valid; it's just worth to mention its limitations or pitfalls. – Julien Carsique Jan 06 '16 at 18:14

score 2 · Answer 5 · answered Nov 04 '11 at 19:10

2

Another short bash solution

variable=$(
  RS=""
  while read line; do
    printf "%s%s" "$RS" "$line"
    RS='%0D%0A'
  done < filename
)

answered Nov 04 '11 at 19:10

glenn jackman

238,783
38
220
352

BiS · Answer 6 · 2017-08-22T07:27:15.190

1

A very simple single-line solution which requires no extra files as its quite easy to understand (I think, just cat the file together and perform sed-replace):

output=$(echo $(cat ./myFile.txt) | sed 's/ /%0D%0A/g')

edited Aug 22 '17 at 07:27

answered Aug 22 '17 at 07:21

BiS

501
4
17

score 1 · Answer 7 · answered Nov 04 '11 at 17:15

awk 'END { print r }
{ r = r ? r OFS $0 : $0 }
  ' OFS='%0D%0A' infile

With shell:

data=

while IFS= read -r; do
  [ -n "$data" ] &&
     data=$data%0D%0A$REPLY ||
    data=$REPLY
done < infile

printf '%s\n' "$data"

Recent bash versions:

data=

while IFS= read -r; do
  [[ -n $data ]] &&
     data+=%0D%0A$REPLY ||
    data=$REPLY
done < infile

printf '%s\n' "$data"

score 0 · Answer 8 · answered Nov 04 '11 at 17:10

0

Useless use of cat, punished! You want to feed the CSV into the loop

while read line; do 
 # ...
done < csv

answered Nov 04 '11 at 17:10

themel

8,825
2
32
31

Getting extra spaces : this,is,some,data%0D%0A this,is,again,data%0D%0A third,line,of,data%0D%0A fourth,line,in,data%0D%0A – Yugal Jindle Nov 04 '11 at 17:33
Interesting, I don't. Are they in the source file or your shell expression maybe? – themel Nov 04 '11 at 17:46

How to concatenate all lines from a file in Bash?

8 Answers8