40

What I'm trying to do is plot the latitude and longitude values of specific storms on a map using matplotlib,basemap,python, etc. My problem is that I'm trying to extract the latitude, longitude, and name of the storms on map but I keep getting errors between lines 41-44 where I try to extract the columns into the list.

Here is what the file looks like:

1957,AUDREY,HU, 21.6N, 93.3W
1957,AUDREY,HU,22.0N,  93.4W
1957,AUDREY,HU,22.6N,  93.5W
1957,AUDREY,HU,23.2N,  93.6W

I want the list to look like the following:

latitude = [21.6N,22.0N,23.4N]
longitude = [93.3W, 93.5W,93.8W]
name = ["Audrey","Audrey"]

Here's what I have so far:

data = np.loadtxt('louisianastormb.csv',dtype=np.str,delimiter=',',skiprows=1)
'''print data'''

data = np.loadtxt('louisianastormb.csv',dtype=np.str,delimiter=',',skiprows=0)

f= open('louisianastormb.csv', 'rb')
reader = csv.reader(f, delimiter=',')
header = reader.next()
zipped = zip(*reader)

latitude = zipped[3]
longitude = zipped[4]
names = zipped[1]
x, y = m(longitude, latitude)

Here's the last error message/traceback I received:

Traceback (most recent call last):
File "/home/darealmzd/lstorms.py", line 42, in

header = reader.next()
_csv.Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?

starball
  • 20,030
  • 7
  • 43
  • 238
mikez1
  • 685
  • 4
  • 9
  • 15
  • 1
    I think the problem is in your csv files not your code. Your code runs as expected with the provided sample csv input. This post may provided some insight. http://stackoverflow.com/questions/6726953/open-the-file-in-universal-newline-mode-using-csv-module-django – RMcG Oct 21 '13 at 05:45

3 Answers3

76

This looks like a problem with line endings in your code. If you're going to be using all these other scientific packages, you may as well use Pandas for the CSV reading part, which is both more robust and more useful than just the csv module:

import pandas
colnames = ['year', 'name', 'city', 'latitude', 'longitude']
data = pandas.read_csv('test.csv', names=colnames)

If you want your lists as in the question, you can now do:

names = data.name.tolist()
latitude = data.latitude.tolist()
longitude = data.longitude.tolist()
chthonicdaemon
  • 19,180
  • 2
  • 52
  • 66
  • this is great. thanks, i hadn't heard of pandas before. one small thing, if you run this on columns with different lengths, the shorter columns will populate with `null`s to match the longest column. you know a fix for this? – Jona Dec 26 '16 at 10:12
  • I'm not sure how you would have columns with different lengths in the CSV? But you can get rid of the na values with [`data.dropna()`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.dropna.html) – chthonicdaemon Dec 26 '16 at 10:21
  • yeah it's a bit of a hack i'm doing for a small script. thanks =] – Jona Dec 26 '16 at 10:44
  • @chthonicdaemon I tried using your code for a similar example, however I think pands doesn't find my file, so it gives me the error " File text.csv does not exist" where instead of text there is the name of my file. Should I add the path of the file? How do I do that? – Euler_Salter Apr 07 '17 at 10:33
  • @Euler_Salter I think that warrants a new question rather than a comment on an old one. – chthonicdaemon Apr 10 '17 at 04:02
  • @chthonicdaemon My CSV file has first row as headings. This method also reads the columns headings in the list. How to avoid reading columns headings into the list. I just want to read the values, so how do I avoid reading first row?? – Rishab Prasad Jun 01 '18 at 14:45
  • @Boy You would just not supply the `names=colnames` part. – chthonicdaemon Jun 02 '18 at 11:06
54

A standard-lib version (no pandas)

This assumes that the first row of the csv is the headers

import csv

# open the file in universal line ending mode 
with open('test.csv', 'rU') as infile:
  # read the file as a dictionary for each row ({header : value})
  reader = csv.DictReader(infile)
  data = {}
  for row in reader:
    for header, value in row.items():
      try:
        data[header].append(value)
      except KeyError:
        data[header] = [value]

# extract the variables you want
names = data['name']
latitude = data['latitude']
longitude = data['longitude']
Ben Southgate
  • 3,388
  • 3
  • 22
  • 31
1
import csv
from sys import argv

d = open("mydata.csv", "r")

db = []

for line in csv.reader(d):
    db.append(line)

# the rest of your code with 'db' filled with your list of lists as rows and columbs of your csv file.
person the human
  • 345
  • 4
  • 15