reading comma separated value from string object in python

Question

I have output from http request which is of string type but the data is like csv. As the output type in my request header is csv ('Accept':"application/csv"). As this the format supported by the source.But the response content type is a string. res=request.contenttype(res)` gives me string.

Here is the sample output from the object(res):

QueryTime
start,end
144488,144490

Data

Data - AData
id,G_id,name,type,time,sid,channel
23,-1,"B1",type1,144488,11,CH23
23,-1,"B1",type1,144488,11,CH23
Data - BData
id,G_id,time,se
23,-1,144488,undefined
23,-1,144488,undefined

If you see the data is in form of csv and there are multiple tables like you see "AData" & "BData" I am not getting which approach to take to read this. I have tried csv module but no help. I have tried dict.csv to convert but again same. Not getting desired output. May be I am doing something wrong as I am new with python. Need is to read each table from the output object.

with open('file.csv', 'wb') as csvfile:
  spamwriter = csv.writer(csvfile, delimiter=',',quoting=csv.QUOTE_NONE)
  spamwriter.writerow(rec)

with open('file.csv') as csvfile:
   reader = csv.DictReader(csvfile)
   for row in reader:
   print row

Experts please guide :-)

`writerow(rec)` is incorrect (`rec` is not a single csv row). Use [`with open('file.csv', 'wb') as file: copyfileobj(r.raw, file)` to save the file if you pass `stream=True` to `requests`](http://stackoverflow.com/a/32814347/4279). — jfs, Oct 15 '15 at 10:43

score 0 · Accepted Answer · answered Oct 15 '15 at 10:50

You could pre-parse the output using a regular expression to extract the various sections, and then use StringIO to parse each section to a csv.reader as follows:

import csv
import StringIO
from collections import OrderedDict

output = """
QueryTime
start,end
144488,144490

Data

Data - AData
id,G_id,name,type,time,sid,channel
23,-1,"B1",type1,144488,11,CH23
23,-1,"B1",type1,144488,11,CH23
Data - BData
id,G_id,time,se
23,-1,144488,undefined
23,-1,144488,undefined"""

sections = ['QueryTime', 'Data - AData', 'Data - BData', 'Data']
re_sections = '|'.join([re.escape(s) for s in sections])
tables = re.split(r'(' + re_sections + ')', output)
tables = [t.strip() for t in tables[1:]]

d_tables = OrderedDict()

for section, table in zip(*[iter(tables)]*2):
    if len(table):
        csv_input = csv.reader(StringIO.StringIO(table))
        d_tables[section] = list(csv_input)

for section, entries in d_tables.items():
    print section
    print entries
    print

Giving you the following output:

QueryTime
[['start', 'end'], ['144488', '144490']]

Data - AData
[['id', 'G_id', 'name', 'type', 'time', 'sid', 'channel'], ['23', '-1', 'B1', 'type1', '144488', '11', 'CH23'], ['23', '-1', 'B1', 'type1', '144488', '11', 'CH23']]

Data - BData
[['id', 'G_id', 'time', 'se'], ['23', '-1', '144488', 'undefined'], ['23', '-1', '144488', 'undefined']]

score 0 · Answer 2 · answered Oct 15 '15 at 11:58

I came up with this function to parse the data:

def parse_data(data):
 parsed = {}
 current_section = None

 for line in data.split('\n'):
  line = line.strip()
  if line:
   if ',' in line:
    current_section.append(line.split(','))
   else:
    parsed[line] = []
    current_section = parsed[line]
 return parsed

It returns a dictionary where each key refers to a section of the input. Its value is a list where each member represents a row of input. Each row is also a list of the individual values as strings. It does not treat the first row in a section specially.

Running it on your input produces this (reformatted for readability):

{
 'Data - AData': [
  ['id', 'G_id', 'name', 'type', 'time', 'sid', 'channel'],
  ['23', '-1', '"B1"', 'type1', '144488', '11', 'CH23'],
  ['23', '-1', '"B1"', 'type1', '144488', '11', 'CH23']
 ],
 'Data - BData': [
  ['id', 'G_id', 'time', 'se'],
  ['23', '-1', '144488', 'undefined'],
  ['23', '-1', '144488', 'undefined']
 ],
 'Data': [
 ],
 'QueryTime': [
  ['start', 'end'],
  ['144488', '144490']
 ]
}

getting an AttributeError: 'module' object has no attribute 'split'. I have added csv module . import csv — Deb, Oct 15 '15 at 13:18
My function `parse_data` takes a regular string (`data`). It does not need any special modules. — Karel Vlk, Oct 16 '15 at 14:23

reading comma separated value from string object in python

2 Answers2