56

I need to remove all the blank lines from an input file and write into an output file. Here is my data as below.

11216,33,1032747,64310,1,0,0,1.878,0,0,0,1,1,1.087,5,1,1,18-JAN-13,000603221321

11216,33,1033196,31300,1,0,0,1.5391,0,0,0,1,1,1.054,5,1,1,18-JAN-13,059762153003

11216,33,1033246,31300,1,0,0,1.5391,0,0,0,1,1,1.054,5,1,1,18-JAN-13,000603211032

11216,33,1033280,31118,1,0,0,1.5513,0,0,0,1,1,1.115,5,1,1,18-JAN-13,055111034001

11216,33,1033287,31118,1,0,0,1.5513,0,0,0,1,1,1.115,5,1,1,18-JAN-13,000378689701

11216,33,1033358,31118,1,0,0,1.5513,0,0,0,1,1,1.115,5,1,1,18-JAN-13,000093737301

11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802041926

11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802041954

11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802049326

11216,33,1035476,37340,1,0,0,1.7046,0,0,0,1,1,1.123,5,1,1,18-JAN-13,045802049383

11216,33,1036985,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000093415580

11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781202001

11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781261305

11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781603955

11216,33,1037003,15151,1,0,0,1.4436,0,0,0,1,1,1.065,5,1,1,18-JAN-13,000781615746
Jonathan Leffler
  • 730,956
  • 141
  • 904
  • 1,278
Teja
  • 13,214
  • 36
  • 93
  • 155
  • This is my sample data and any of the below commands dont work for me. :( – Teja Jan 28 '13 at 20:34
  • 1
    They didn't work **because you asked the wrong question**. Did you actually try `sed -i '/^[[:space:]]*$/d' foo`? Because if that didn't work then you need to restate the problem – Jonathan Wakely Jan 28 '13 at 20:42
  • @SOaddict, I see multiple answers here that answer your original question as well as ones that handle whitespace in the "blank" lines. If those don't work, I think you need to examine your input file and update the question. – gpojd Jan 28 '13 at 20:51
  • I copy pasted the exact input file sample. – Teja Jan 28 '13 at 21:19
  • check this:http://theunixshell.blogspot.com/2013/01/deleting-empty-lines-from-file.html – Vijay Jan 29 '13 at 10:12
  • 1
    One possible reason why the commands below, which should work, seemed not to work is that the file originated on Windows or transited via a Windows machine and actually has CRLF (carriage return, line feed) endings instead of Unix-style NL (newline, aka LF) line endings. Unix tools treat the CR as just another character, and therefore regexes like `/^$/` would fail to match a blank line with a CRLF ending. The version using `sed -i '/^[[:space:]]*$/d;s/[[:space:]]*$//'` should work for CRLF lines, too; the `/^[[:space:]]*$/` pattern matches a CR in the line. – Jonathan Leffler Mar 31 '15 at 14:36

8 Answers8

97
sed -i '/^$/d' foo

This tells sed to delete every line matching the regex ^$ i.e. every empty line. The -i flag edits the file in-place, if your sed doesn't support that you can write the output to a temporary file and replace the original:

sed '/^$/d' foo > foo.tmp
mv foo.tmp foo

If you also want to remove lines consisting only of whitespace (not just empty lines) then use:

sed -i '/^[[:space:]]*$/d' foo

Edit: also remove whitespace at the end of lines, because apparently you've decided you need that too:

sed -i '/^[[:space:]]*$/d;s/[[:space:]]*$//' foo
Jonathan Wakely
  • 166,810
  • 27
  • 341
  • 521
  • 8
    I considered that possibility so wrote the "if your sed doesn't support that" part. Seems like at least seven people have wasted their time trying to help you and you either aren't reading or aren't thinking properly. – Jonathan Wakely Jan 28 '13 at 21:33
  • Error msg: sed: illegal option -- i Usage: sed [-n] [-e script] [-f source_file] [file...] – Teja Jan 28 '13 at 21:34
  • so you were doing it wrong for 19 hours by not reading properly or not using copy'n'paste properly? sheesh – Jonathan Wakely Jan 29 '13 at 15:37
  • 1
    It was my bad Jonathan.Your answer was right. Didn't check it again from yesterday. – Teja Jan 29 '13 at 15:39
  • 3
    anyone who has trouble with the -i option on a Mac: use `-i .bak` instead (as in `sed -i .bak '/^[[:space:]]*$/d' foo`. The version of sed we have wants a file extension to use when doing in-place edits. So passing `-i .bak` tells it to copy the existing file with an extension of `.bak` that you can restore incase the in-place edit fails. – gMale Feb 22 '15 at 07:44
  • 1
    Note that the `-i` option in GNU `sed` also accepts a backup suffix as an optional argument, so if you write `-i.bak` with the option and argument touching, the code will work with both BSD (Mac OS X) `sed` and GNU `sed`. Mac OS X requires the backup suffix and allows it to be separate from the `-i` option. Standards: such wonderful things… – Jonathan Leffler Mar 31 '15 at 14:30
45
awk 'NF' filename


awk 'NF > 0' filename


sed -i '/^$/d' filename


awk '!/^$/' filename


awk '/./' filename

The NF also removes lines containing only blanks or tabs, the regex /^$/ does not.

Jotne
  • 40,548
  • 12
  • 51
  • 55
  • In case someone have confusion > https://stackoverflow.com/questions/23544804/how-awk-nf-filename-is-working – jian Jan 13 '22 at 10:52
17

Use grep to match any line that has nothing between the start anchor (^) and the end anchor ($): 

grep -v '^$' infile.txt > outfile.txt

If you want to remove lines with only whitespace, you can still use grep. I am using Perl regular expressions in this example, but here are other ways: 

grep -P -v '^\s*$' infile.txt > outfile.txt

or, without Perl regular expressions: 

grep -v '^[[:space:]]*$' infile.txt > outfile.txt
gpojd
  • 22,558
  • 8
  • 42
  • 71
12
sed -e '/^ *$/d' input > output

Deletes all lines which consist only of blanks (or is completely empty). You can change the blank to [ \t] where the \t is a representation for tab. Whether your shell or your sed will do the expansion varies, but you can probably type the tab character directly. And if you're using GNU or BSD sed, you can do the edit in-place, if that's what you want, with the -i option.

If I execute the above command still I have blank lines in my output file. What could be the reason?

There could be several reasons. It might be that you don't have blank lines but you have lots of spaces at the end of a line so it looks like you have blank lines when you cat the file to the screen. If that's the problem, then:

sed -e 's/  *$//' -e '/^ *$/d' input > output

The new regex removes repeated blanks at the end of the line; see previous discussion for blanks or tabs.

Another possibility is that your data file came from Windows and has CRLF line endings. Unix sees the carriage return at the end of the line; it isn't a blank, so the line is not removed. There are multiple ways to deal with that. A reliable one is tr to delete (-d) character code octal 15, aka control-M or \r or carriage return:

tr -d '\015' < input | sed -e 's/  *$//' -e '/^ *$/d' > output

If neither of those works, then you need to show a hex dump or octal dump (od -c) of the first two lines of the file, so we can see what we're up against:

head -n 2 input | od -c

Judging from the comments that sed -i does not work for you, you are not working on Linux or Mac OS X or BSD — which platform are you working on? (AIX, Solaris, HP-UX spring to mind as relatively plausible possibilities, but there are plenty of other less plausible ones too.)

You can try the POSIX named character classes such as sed -e '/^[[:space:]]*$/d'; it will probably work, but is not guaranteed. You can try it with:

echo "Hello World" | sed 's/[[:space:]][[:space:]]*/   /'

If it works, there'll be three spaces between the 'Hello' and the 'World'. If not, you'll probably get an error from sed. That might save you grief over getting tabs typed on the command line.

Jonathan Leffler
  • 730,956
  • 141
  • 904
  • 1,278
8
grep . file

grep looks at your file line-by-line; the dot . matches anything except a newline character. The output from grep is therefore all the lines that consist of something other than a single newline.

jscs
  • 63,694
  • 13
  • 151
  • 195
Ed Morton
  • 188,023
  • 17
  • 78
  • 185
  • 1
    You might use `grep '[^[:space:]]' file` in case the "blank" lines contain whitespace. – glenn jackman Jan 29 '13 at 00:59
  • 5
    Would 19 characters of explanation really have been too much? – jscs Jan 29 '13 at 01:32
  • 1
    @JoshCaswell It'd just be pointless. It'd be like when you read code that says "i=0" and someone adds a comment "set the variable i to the value zero" because someone sometime told them that adding comments to code is a good idea. In this case either it's glaringly obvious what that grep command does or the OP REALLY needs to read the man page. – Ed Morton Jan 29 '13 at 14:22
  • 2
    Less pointless than a copy-pasted error message about your post being too short? We have different definitions of "pointless". – jscs Mar 27 '13 at 18:49
  • Not less pointless, equally pointless but more effort. In a perfect world the web site simply wouldn't require a minimum number of characters and then we wouldn't have had this fruitless discussion. – Ed Morton Mar 27 '13 at 19:15
7

with awk

awk 'NF > 0' filename

Mirage
  • 30,868
  • 62
  • 166
  • 261
3

To be thorough and remove lines even if they include spaces or tabs something like this in perl will do it:

cat file.txt | perl -lane "print if /\S/"

Of course there are the awk and sed equivalents. Best not to assume the lines are totally blank as ^$ would do.

Cheers

G. Cito
  • 6,210
  • 3
  • 29
  • 42
2

You can sed's -i option to edit in-place without using temporary file:

 sed -i '/^$/d' file
P.P
  • 117,907
  • 20
  • 175
  • 238
  • Then the lines are not "blank" lines. Jonathan Wakely answered how to remove whitespaces too. Doesn't it work? – P.P Jan 28 '13 at 21:24
  • Can you describe it clearly how exactly the blank lines are still there? – P.P Jan 28 '13 at 21:32
  • My 19th column has whitespaces.So I guess I should remove whitespaces first and then execute any of these commands.. – Teja Jan 28 '13 at 21:33
  • No. `sed -i '/^[[:space:]]*$/d' file` should remove all lines with any whitespace such as , , or any combination of these. – P.P Jan 28 '13 at 21:37
  • There's no syntax error in the command. Do you use it *exacty* like in my previous comment? can you copy-paste the exact command you use? – P.P Jan 28 '13 at 21:40
  • sed: illegal option -- i Usage: sed [-n] [-e script] [-f source_file] [file...] – Teja Jan 28 '13 at 21:44
  • No. How you use the sed command. Is it like this? `sed -i '/^[[:space:]]*$/d' filename` ? – P.P Jan 28 '13 at 21:46
  • Yes its the same way as it is mentioned in the answer. – Teja Jan 28 '13 at 21:47
  • You cannot avoid a temporary file with `sed -i`, because `sed -i` uses a temporary file. – William Pursell Jan 28 '13 at 23:00
  • Right. At least saving externally moving again is not needed. It seems `-i` option is not supported on OP's system. Though there are working solutions here, he just doesn't seem to get it working! – P.P Jan 28 '13 at 23:07