How can I replace multiple empty lines with a single empty line in bash?

Question

I have a file that contains:

something



something else

something else again

I need a bash command, sed/grep w.e that will produce the following output

something

something else

something else again

In other words, I need to replace multiple blank lines with just a single blank line. grep/sed are line based. I've never found a BASH solution that would work on multi-line regex patterns.

Quick clarification question: Does the file ever have lines that *aren't* separated by a blank line? — biggusjimmus, May 28 '09 at 18:28
Do you ever have duplicate lines that are not blank? If not, you could use uniq. — Nathan Kitchen, May 28 '09 at 18:38
There are lines that are not separated by a blank line. There could be duplicate lines. — Nick Zalutskiy, May 28 '09 at 20:26

score 121 · Accepted Answer · edited Jul 30 '17 at 22:44

121

For BSD-derived systems (including GNU):

You just need cat with the -s option which causes it to remove repeated empty lines from its output:

cat -s

From man page: -s --squeeze-blank: suppress repeated empty output lines.

edited Jul 30 '17 at 22:44

o11c

15,265
4
50
75

answered Jun 01 '09 at 11:32

marco

4,455
1
23
20

7

for whatever reason that doesnt work for me `cat -s file1 > file2` – qodeninja Mar 26 '14 at 03:20
5

@qodeninja It didn't work for me either when I was working with files with Windows line endings. Maybe that could be the problem for you too? When I converted the files to Unix line endings `cat -s` worked fine for me. – Victor Zamanian Apr 02 '14 at 19:40
in my version of `cat` on Solaris, the `-s` option has a different meaning "-s cat is silent about non-existent files" – dokaspar Mar 26 '15 at 13:23
@VictorZamanian Most of the time it is not possible to redirect back into a file like qodeninja did. For more information, read here: http://stackoverflow.com/questions/6696842/bash-redirect-input-from-file-back-into-same-file – Filip Allberg Nov 30 '16 at 18:47
@FilipAllberg Did they? I see `file1` and `file2`. – Victor Zamanian Nov 30 '16 at 19:03
3

Note that it can be used through a pipe : `some_output|cat -s`. – Skippy le Grand Gourou Mar 22 '17 at 15:44
All other answers should be deleted, this is the only one you will ever need. (j/k) – sjas Oct 31 '17 at 10:41
I would like to use it in my mailcap file on the output of links2 but it does not seem to change anything. `text/html; links2 -g -dump %s | cat -s ; nametemplate=%s.html; copiousoutput` – djangonaut Apr 06 '18 at 18:20
GNU systems aren't BSD-derived. – Michael-O Sep 18 '20 at 09:16
trailing newlines not work. – John Chen Oct 31 '21 at 04:47

score 40 · Answer 2 · edited Jan 23 '21 at 07:36

40

I just solved this problem by sed. Even if this is a 7 year old question, someone may find this helpful, so I am writing my solution by sed here:

sed 'N;/^\n$/D;P;D;'

edited Jan 23 '21 at 07:36

Dana Robinson

4,304
8
33
41

answered Jul 22 '16 at 07:06

WKPlus

6,955
2
35
53

3

This works well for me, and because it's sed, I can use -i for in place fixes. – Kieran Jun 13 '17 at 11:57
2

Wierdly this also removes the last 2 lines fo the file for me (on OSX sed) – Nicolas Garnier Feb 27 '18 at 11:41
3

`sed '$!N;/^\n$/{$q;D;};P;D;'` prevents the last line from being deleted on OSX sed. Also works on gnu sed. – parleer Apr 01 '20 at 17:04
1

Thanks, just so you know this is still useful info. – Jeff Adamson Jul 23 '20 at 21:08

score 17 · Answer 3 · edited Jul 22 '16 at 18:15

17

grep -A1 . <yourfile> | grep -v "^--$"

This grep solution works assuming you want the following:

Input

line1

line2
line3


line4



line5

Output

line1

line2
line3

line4

line5

edited Jul 22 '16 at 18:15

heemayl

39,294
7
70
76

answered May 28 '09 at 18:45

biggusjimmus

2,706
3
26
31

I like it, very elegant solution – duckyflip May 29 '09 at 00:24
5

Frickin' brilliant, actually. I wouldn't have come up with this in a million years. Nice work. A _slightly_ more robust version goes like this of course (to handle non-empty blank lines): grep -v -A1 '^[[:blank:]]*$' | grep -v '^--$' – Dan Moulding May 29 '09 at 03:43

score 13 · Answer 4 · answered May 28 '09 at 18:37

13

Actually, if you replace multiple newlines with a single newline, the output would be:

something
something else
something else again

You can achieve this by:

sed /^$/d FILE

answered May 28 '09 at 18:37

Can Berk Güder

109,922
25
130
137

This is correct, obviously :) What OP likely meant is to replace multiple "blank" (note that this is not necessarily empty, since lines can have invisible white space) with one empty line (which unless it is at the beginning or end of the sequence) means two newline chars. – arielf Jun 25 '16 at 20:59

score 4 · Answer 5 · answered May 27 '13 at 17:18

4

If someone want use perl

perl -00pe0 < file

will do the same, as cat -s :)

answered May 27 '13 at 17:18

clt60

62,119
17
107
194

Shaoyun · Answer 6 · 2019-12-05T08:25:05.913

4

Use awk:

awk '{ /^\s*$/?b++:b=0; if (b<=1) print }' file

Breakdown:

/^\s*$/?b++:b=0
    - ? :       the ternary operator
    - /^\s*$/   matches a blank line
    - b         variable that counts consecutive blank lines (b++).
                however, if the current line is non-blank, b is reset to 0.


if (b<=1) print
    print if the current line is non-blank (b==0)
          or if there is only one blank line (b==1).

By adjusting the regex, you can generalize it to other scenarios like squeezing multiple blank lines (">") in email: https://stackoverflow.com/a/59189823/12483961

edited Dec 05 '19 at 08:25

answered Dec 05 '19 at 08:19

Shaoyun

91
4

1

I really like this, because it is the first solution that works on DOS files (with \r\n newlines) and because you can change the b<=1 to, say, b<=2 to allow up to 2 blank lines. Btw you can replace the file inline with: awk -i inplace – JohnFlux Sep 30 '21 at 22:40

score 4 · Answer 7 · answered May 28 '09 at 18:41

4

A solution with awk, which replaces several blank lines with a single blank line:

awk 'BEGIN{bl=0}/^$/{bl++;if(bl==1)print;else next}/^..*$/{bl=0;print}' myfile

answered May 28 '09 at 18:41

mouviciel

66,855
13
106
140

score 4 · Answer 8 · answered May 28 '09 at 18:42

4

Usually, if I find that sed can't do something I need, I turn to awk:

awk '
BEGIN {
    blank = 0;
}

/^[[:blank:]]*$/ {
     if (!blank) {
          print;
     }
     blank = 1;
     next;
}

{
     print;
     blank = 0;
}' file

answered May 28 '09 at 18:42

Dan Moulding

211,373
23
97
98

score 3 · Answer 9 · edited May 23 '17 at 12:18

3

This uses marco's solution on multiple files:

for i in *; do FILE=$(cat -s "$i"); echo "$FILE" > "$i"; done

edited May 23 '17 at 12:18

Community

1
1

answered Jul 26 '13 at 16:54

score 2 · Answer 10 · answered Jul 22 '16 at 18:24

2

Python, with regular expression:

import re
import sys
sys.stdout.write(re.sub('\n{2,}','\n\n', sys.stdin.read()))

answered Jul 22 '16 at 18:24

Robᵩ

163,533
20
239
308

score 2 · Answer 11 · answered May 28 '09 at 18:30

2

Use python:

s = file("filename.txt").read()
while "\n\n\n" in s: s = s.replace("\n\n\n", "\n\n")
import sys
sys.stdout.write(s)

answered May 28 '09 at 18:30

Jonas Kölker

7,680
3
44
51

1

this is inefficient while loop – user5672998 Dec 15 '17 at 09:49

score 0 · Answer 12 · answered May 28 '09 at 18:35

0

Super easy to do with vim. Just open the file and type the following:

:%s/\n\n\n*/\r\r/

That will reduce all blocks of more than 2 new lines to 2 new lines. Hope this helps!

answered May 28 '09 at 18:35

Carl Sverre

1,139
10
19

score -3 · Answer 13 · answered May 28 '09 at 18:41

-3

I take it that you'll probably want to remove lines that only have whitespace.

That can be done with:

sed /^[:space:]*$/d FILE

answered May 28 '09 at 18:41

Clue Less

3,147
2
20
10

score -6 · Answer 14 · answered May 31 '09 at 14:57

-6

Pipelining it to |uniq may be solution (if other than empty lines don't duplicate)

answered May 31 '09 at 14:57

mateusza

5,341
2
24
20

How can I replace multiple empty lines with a single empty line in bash?

14 Answers14

Linked