Concatenating Files And Insert New Line In Between Files

Question

I have multiple files which I want to concat with cat. Let's say

File1.txt 
foo

File2.txt
bar

File3.txt
qux

I want to concat so that the final file looks like:

foo

bar

qux

Instead of this with usual cat File*.txt > finalfile.txt

foo
bar 
qux

What's the right way to do it?

possible duplicate of [How do I include a blank line between files I'm concatenating with "cat"?](http://stackoverflow.com/questions/1653063/how-do-i-include-a-blank-line-between-files-im-concatenating-with-cat) — legoscia, Mar 19 '13 at 18:42

codaddict · Accepted Answer · 2011-11-18T13:48:41.143

183

You can do:

for f in *.txt; do (cat "${f}"; echo) >> finalfile.txt; done

Make sure the file finalfile.txt does not exist before you run the above command.

If you are allowed to use awk you can do:

awk 'FNR==1{print ""}1' *.txt > finalfile.txt

edited Nov 18 '11 at 13:48

answered Nov 18 '11 at 13:36

codaddict

445,704
82
492
529

24

`AWK '{print $0}' *.txt` – timger Jul 13 '15 at 03:20
9

This has the distinct flaw that there will be empty lines either at the end (from the first alternative) or in the beginning (second alternative). You can easily guard against this with `awk 'FNR==1 && NR > 1 ...'` instead, though. – tripleee Feb 16 '16 at 04:58
6

If you put `>finalfile.txt` after the `done` you can overwrite instead of append, which will remove the requirement to make sure the file is missing or empty before the loop. – tripleee Feb 16 '16 at 04:59
2

Inspired by @timger, `awk '1' *.txt` :smile: – hustnzj Oct 09 '22 at 02:27

score 78 · Answer 2 · answered May 08 '14 at 18:32

78

If you have few enough files that you can list each one, then you can use process substitution in Bash, inserting a newline between each pair of files:

cat File1.txt <(echo) File2.txt <(echo) File3.txt > finalfile.txt

answered May 08 '14 at 18:32

Robert Tupelo-Schneck

10,047
4
47
58

1

Beautiful! Thanks. – Bob Kocisko Jan 07 '17 at 22:38
This worked nicely for me for my Letsencrypt certs to create .pem files. – leeman24 Jul 10 '19 at 17:01
I'm trying to do it inside a xargs and this fails, any hint? `… | xargs -I{} kubectl -n alex exec {} -- cat blah.log <(echo) >> blahblah.logs` cat: /dev/fd/63: No such file or directory, command terminated with exit code 1 – tuxErrante May 04 '21 at 09:04
1

@tuxErrante I think the problem is not xargs but kubectl, which is taking a command to run remotely, while `<(echo)` is running locally. Maybe `-- bash -c "cat blah.log <(echo)"`? – Robert Tupelo-Schneck May 04 '21 at 18:17
It dind't work on an alpine shell, but the following did, thanks for the hint! `sh -c "echo -e '\n\n' | cat - /../logs/a.log" ` – tuxErrante May 05 '21 at 10:50

score 41 · Answer 3 · edited Aug 01 '21 at 04:09

41

If it were me doing it I'd use sed:

sed -e '$s/$/\n/' -s *.txt > finalfile.txt

In this sed pattern $ has two meanings, firstly it matches the last line number only (as a range of lines to apply a pattern on) and secondly it matches the end of the line in the substitution pattern.

If your version of sed doesn't have -s (process input files separately) you can do it all as a loop though:

for f in *.txt ; do sed -e '$s/$/\n/' $f ; done > finalfile.txt

edited Aug 01 '21 at 04:09

answered Nov 18 '11 at 13:41

Flexo

87,323
22
191
272

4

Or in GNU sed: `sed -s '$G' *.txt > finalfile.txt` – Ruud Helderman May 16 '16 at 06:53
1

Only one stream! this should be the accepted answer! – Yassine ElBadaoui Feb 23 '18 at 11:21
1

be careful guys, I just crashed my PC because I used `find` in place of the `*.txt`, which meant the file was appended onto itself! – xeruf Jun 02 '18 at 15:54

score 17 · Answer 4 · answered Nov 10 '16 at 19:56

17

This works in Bash:

for f in *.txt; do cat $f; echo; done

In contrast to answers with >> (append), the output of this command can be piped into other programs.

Examples:

for f in File*.txt; do cat $f; echo; done > finalfile.txt
(for ... done) > finalfile.txt (parens are optional)
for ... done | less (piping into less)
for ... done | head -n -1 (this strips off the trailing blank line)

answered Nov 10 '16 at 19:56

teichert

3,963
1
31
37

`for f in *; do echo -e "# ------ $f ------"; cat $f; echo -e "\n"; done` also print file name. – Eric Mar 16 '22 at 04:35

score 14 · Answer 5 · answered Jun 21 '17 at 11:56

14

You may do it using xargs if you like, but the main idea is still the same:

find *.txt | xargs -I{} sh -c "cat {}; echo ''" > finalfile.txt

answered Jun 21 '17 at 11:56

Nick Roz

3,918
2
36
57

3

Thanks. I find `xargs` much easier to use than loops in bash. – RawwrBag Jan 18 '18 at 18:22

score 10 · Answer 6 · edited Jul 29 '21 at 16:24

10

That's how I just did it on OsX 10.10.3

for f in *.txt; do (cat $f; echo '') >> fullData.txt; done

since the simple 'echo' command with no params ended up in no new lines inserted.

edited Jul 29 '21 at 16:24

Max Alibaev

681
7
17

answered Aug 13 '15 at 14:22

lbrutti

1,181
13
19

This puts the string at the end of the file; how do I get it inserted between _each_ file? – onassar Nov 19 '18 at 16:37

score 5 · Answer 7 · answered Aug 04 '22 at 02:15

5

You could use grep, with -h to not echo the filenames

grep -h "" File*.txt

Will give:

foo
bar 
qux

answered Aug 04 '22 at 02:15

Wadih M.

12,810
7
47
57

This command is the simplest and easy. Thanks. – Shezan Jun 12 '23 at 09:59

score 3 · Answer 8 · edited Jul 29 '21 at 14:49

3

In python, this concatenates with blank lines between files (the , suppresses adding an extra trailing blank line):

print '\n'.join(open(f).read() for f in filenames),

Here is the ugly python one-liner that can be called from the shell and prints the output to a file:

python -c "from sys import argv; print '\n'.join(open(f).read() for f in argv[1:])," File*.txt > finalfile.txt

edited Jul 29 '21 at 14:49

Max Alibaev

681
7
17

answered Nov 10 '16 at 20:12

teichert

3,963
1
31
37

score 0 · Answer 9 · answered Aug 24 '23 at 08:10

A POSIX compatible solution is to use cat, but interlude each file with a file containing only a blank line.

nl=`mktemp`
printf '\n' > $nl
cat file1 $nl file2 $nl file3
rm $nl

A more fancy version might be something line.

nl=`mktemp`
printf '\n' > $nl
find file1 file2 file3 -print0 |
  xargs -0 printf "%s\0$nl\0" |
  tr '\0' '\n' |
  sed -n '$!p;$q' |
  tr '\n' '\0' |
  xargs -0 cat
rm $nl

Concatenating Files And Insert New Line In Between Files

9 Answers9

Linked

Related