CSV Returning as Bytes not String Python

Question

The error I am recieving is this:

_csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)

Where in my code would I open the file in text mode?

import csv
import urllib.request
import pymysql.cursors
connection = pymysql.connect(host='localhost',
                         user='',
                         password='',
                         db='mydb',
                         charset='utf8',
                         cursorclass=pymysql.cursors.DictCursor)
try:
    url = 'https://api.iextrading.com/1.0/stock/market/collection/sector?
    collectionName=Health%20Care&format=csv'
    response =  urllib.request.urlopen(url)
    csv_data = csv.reader(response)
    for row in csv_data:

    cursor.execute('INSERT INTO Financials (names, \
          classes, mark )' \
          'VALUES("%s", "%s", "%s")', 
          row)
finally:
    connection.close()

you could slightly fix your indentation as well – Jean-François Fabre Oct 20 '18 at 09:32 — Jean-François Fabre, Oct 20 '18 at 09:32

Jean-François Fabre · Accepted Answer · 2018-10-20T09:40:23.607

response =  urllib.request.urlopen(url)

here response returns an iterator on a bytes object,.

If you know that the csv file is just plain text, you could just insert a generator comprehension to decode the lines:

csv_data = csv.reader(line.decode() for line in response)

or map (don't you love python 3 map when you can use it without lambda?)

csv_data = csv.reader(map(bytes.decode,response))

self-contained example:

import urllib.request,csv

url = 'https://api.iextrading.com/1.0/stock/market/collection/sector?collectionName=Health%20Care&format=csv'
response =  urllib.request.urlopen(url)
for row in csv.reader(line.decode() for line in response):
    print(row)

now you're feeding csv.reader with a list of strings, which will work.

sample output:

['CAPR', 'Capricor Therapeutics Inc.', 'NASDAQ Capital Market', 'Healthcare', 'close', '0.9011', '1539955800519', '0.875', '1539979200341', '0.9011', '0.8702', '0.875', 'Close', 'October 19, 2018', '1539979200341', '96625', '', '', '', '0.875', '1539979200341', '0.9011', '0.0261', '0.02983', '1539954158310', '0.9011', '-0.0261', '-0.02896', '', '', '106532', '', '', '', '', '26905263', '-1.72', '3.19', '0.851', '-0.4716351592356688']
['AVDL', 'Avadel Pharmaceuticals plc', 'NASDAQ Global Market', 'Healthcare', 'close', '4.24', '1539955800643', '4.1', '1539979200312', '4.24', '4', '4.1', 'Close', 'October 19, 2018', '1539979200312', '78386', '', '', '', '4.1', '1539979200312', '4.2', '0.1', '0.02439', '1539892803066', '4.2', '-0.1', '-0.02381', '', '', '114387', '', '', '', '', '150734712', '-3.8', '11.93', '3.98', '-0.5579009090909092']

Note that this will blindly decode the response content with the utf-8 codec, which might not match the actual encoding. Most servers will include the correct encoding in the HTTP headers. There's a question about finding the correct response encoding [here](https://stackoverflow.com/q/4981977/1222951), but none of the answers there are really good. — Aran-Fey, Oct 20 '18 at 09:57
@Aran-Fey correct. In that case that works (tested with the real site). This encoding guess game is tough. — Jean-François Fabre, Oct 20 '18 at 10:29

CSV Returning as Bytes not String Python

1 Answers1