How to skip blank lines with read_fwf in pandas?

Question

I use pandas.read_fwf() function in Python pandas 0.19.2 to read a file fwf.txt that has the following content:

# Column1 Column2
      123     abc

      456     def

#
#

My code is the following:

import pandas as pd
file_path = "fwf.txt"
widths = [len("# Column1"), len(" Column2")]
names = ["Column1", "Column2"]
data = pd.read_fwf(filepath_or_buffer=file_path, widths=widths, 
                   names=names, skip_blank_lines=True, comment="#")

The printed dataframe is like this:

    Column1 Column2
0   123.0   abc
1   NaN     NaN
2   456.0   def
3   NaN     NaN

It looks like the skip_blank_lines=True argument is ignored, as the dataframe contains NaN's.

What should be the valid combination of pandas.read_fwf() arguments that would ensure the skipping of blank lines?

Try taking out the `comment` argument, it might be overriding the `skip_blank_lines` — rassar, Mar 03 '17 at 15:38
@rassar That was my thought too, but it doesn't seem to help. — miradulo, Mar 03 '17 at 15:38
@Mitch Hmm...in the docs (http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_fwf.html) it says `skip_blank_lines` defaluts to `True`...maybe take that one out? — rassar, Mar 03 '17 at 15:40
@rassar That won't help.. `True is True` - OP is just being explicit :) — miradulo, Mar 03 '17 at 15:41
Not sure how blank lines are defined for pandas. Perhaps it only filters `\n` but not other whitespace?. One post-parsing option would be to drop `NaN` rows with `data.dropna(how="all", inplace=True) ` — ODiogoSilva, Mar 03 '17 at 15:44
[Duplicate ?](http://stackoverflow.com/questions/11605909/blank-lines-skip-true-fails-with-read-fwf) — B. M., Mar 03 '17 at 16:14
Yes it seems to be a bug that `read_fwf` is disregarding `skip_blank_lines`. You can get a message when NA values are inferred by enabling `verbose=True`. You could report this bug as a [pandas issue](https://github.com/pandas-dev/pandas/issues). Also you can disable NAs being inferred on blank lines with `na_filter = None` — smci, Sep 03 '18 at 00:30

unutbu · Answer 1 · 2017-03-03T20:30:31.893

import io
import pandas as pd
file_path = "fwf.txt"
widths = [len("# Column1 "), len("Column2")]
names = ["Column1", "Column2"]

class FileLike(io.TextIOBase):
    def __init__(self, iterable):
        self.iterable = iterable
    def readline(self):
        return next(self.iterable)

with open(file_path, 'r') as f:
    lines = (line for line in f if line.strip())
    data = pd.read_fwf(FileLike(lines), widths=widths, names=names, 
                       comment='#')
    print(data)

prints

   Column1 Column2
0      123     abc
1      456     def

with open(file_path, 'r') as f:
    lines = (line for line in f if line.strip())

defines a generator expression (i.e. an iterable) which yields lines from the file with blank lines removed.

The pd.read_fwf function can accept TextIOBase objects. You can subclass TextIOBase so that its readline method returns lines from an iterable:

class FileLike(io.TextIOBase):
    def __init__(self, iterable):
        self.iterable = iterable
    def readline(self):
        return next(self.iterable)

Putting these two together gives you a way to manipulate/modify lines of a file before passing them to pd.read_fwf.

How to skip blank lines with read_fwf in pandas?

1 Answers1

Linked