Getting error when trying to stream a file through csv.DictReader

Question

I have the following route in my Flask app, where I'm basically trying to stream an uploaded TSV file (flask.request.files['file']) and then iterate through its contents. In my code below, it worked fine in Python 2, but now that I've changed to Python 3, I get the error:

  File "/Users/cdastmalchi/Desktop/author_script/main.py", line 89, in process_file
    for line in contents:
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/csv.py", line 111, in __next__
    self.fieldnames
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/csv.py", line 98, in fieldnames
    self._fieldnames = next(self.reader)
_csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)

I also tried doing this but it didn't work either because of TypeError: expected str, bytes or os.PathLike object, not FileStorage:

io.open(file, mode='r', buffering=-1, encoding=None, errors=None, newline=None, closefd=True)

Question: How can I properly stream the input file so that I can iterate through each line?

Flask code

@app.route('/process_file', methods=['POST'])
def process_file():

    # Run checks on the file
    if 'file' not in flask.request.files or not flask.request.files['file'].filename:
        return flask.jsonify({'result':'False', 'message':'no files selected'})
        return flask.redirect(url_for('home'))
    file = flask.request.files['file']

    # Stream file and check that places exist
    contents = csv.DictReader(file, delimiter='\t')
    for line in contents:
        print(line)
    return None

Update

I've tried the following variations of using StringIO as well, but it still comes up blank or yields the same error of _csv.Error: iterator should return strings, not bytes (did you open the file in text mode?):

contents = csv.DictReader(str(file.read()), delimiter='\t')

contents = csv.DictReader(io.StringIO(str(file.read())), delimiter='\t')

contents = csv.DictReader(io.StringIO(file.read()), delimiter='\t')

contents = csv.DictReader(file.read().splitlines(), delimiter='\t')

csv_io = io.StringIO(str(file.read()))
csv_io.seek(0)

contents = csv.DictReader(csv_io, delimiter='\t')

Possible duplicate of [Read file data without saving it in Flask](https://stackoverflow.com/questions/20015550/read-file-data-without-saving-it-in-flask) — djnz, Jul 17 '19 at 23:14

score 2 · Answer 1 · answered Apr 28 '20 at 18:01

I was running into the same issue and getting the same error. I'm not sure if this will help in your situation, but I thought I'd add it here just in case it does.

I believe my problem was that some of the strings in the file were encoded. I was able to fix the problem by adding a .decode() to the file string.

Here's my code:

f = request.files.get['data_file']
file_str = f.read()

# Below is the decode() that fixed the error
for row in csv.DictReader(file_str.decode().splitlines(), skipinitialspace=True):
    data += [{k: v for k, v in row.items()}]

This code converts my uploaded .csv file into a list of dictionaries, each row being its own item in the list. I hope that helps!

Getting error when trying to stream a file through csv.DictReader

1 Answers1