Get dictionary from csv file by conversion

Question

I'm trying to convert a csv to a dict, but I can't get the right output.

Basically I have a csv file which contains integers in the 6th and 7th column, which should be converted in 'lat' and 'lon' in the dict.

This is what I'm trying:

with open('secondhand_data.csv', 'rb') as f:
    reader = csv.reader(f, delimiter=';')
    output2['lon'] = {row[6] for row in reader}
    output2['lat'] = {row[7] for row in reader}

My result is:

{'lat': set([]), 'lon': set(['', '16.33052', '16.38002', '16.38414', '16.34794', '16.34797', '16.33351', '16.46922', '16.33353', '16.33367', '16.38552', '16.33354', '16.41716', ...])}

But what I need is precisely:

{'lat': ['53.1445116550943', '53.134787053494', '53.1383785260816', '53.330366', '55.8611098159417', ...], 'lon' : ['', '16.33052', '16.38002', '16.38414', '16.34794', '16.34797', '16.33351', ...]}

Does anyone know what I'm doing wrong? Many thanks in advance!

@jacoblauw: the csv file looks like this... its a bit messy, sorry for that.

Waage, Deko/ Vintage/ Retro/ Nostalgie;http://www.willhaben.at/iad/kaufen-und-verkaufen/d/waage-deko-vintage-retro-nostalgie-208813335/;"
Doris S.";10;1030 Wien, 03. Bezirk, Landstraße; ;16.38575;48.19149;Fasangasse, Ecke Rennweg1030 Wien, 03. Bezirk, LandstraßeWien
schloss;http://www.willhaben.at/iad/kaufen-und-verkaufen/d/schloss-208789390/;"
Angelina";10;1100 Wien, 10. Bezirk, Favoriten; ;16.35809;48.17209;Hardtmuthgasse1100 Wien, 10. Bezirk, FavoritenWien
RC Boot;http://www.willhaben.at/iad/kaufen-und-verkaufen/d/rc-boot-208786703/;"
privat";15;1100 Wien, 10. Bezirk, Favoriten; ;;;
taschenmesser solingen dirlam & Sohn Söhne horn kleine beschädigung mit hülle;http://www.willhaben.at/iad/kaufen-und-verkaufen/d/taschenmesser-solingen-dirlam-sohn-soehne-horn-kleine-beschaedigung-mit-huelle-208768953/;"
rabe";25;1200 Wien, 20. Bezirk, Brigittenau; ;16.37349;48.23059;brigittenau1200 Wien, 20. Bezirk, BrigittenauWien
Konvolut Gama Norev Matchbox Lesney Corgi Wiking Schuco piccolo...14 teilig, alt, defekt;http://www.willhaben.at/iad/kaufen-und-verkaufen/d/konvolut-gama-norev-matchbox-lesney-corgi-wiking-schuco-piccolo-14-teilig-alt-defekt-208752031/;"
Peter";50;1190 Wien, 19. Bezirk, Döbling; ;16.35537;48.27574;Heiligenstädterstr.1190 Wien, 19. Bezirk, DöblingWien
DAS HAUS DER SCHWÄNE/ A. J. CRONIN, geb. Jubiläumsausgabe von 1934,;http://www.willhaben.at/iad/kaufen-und-verkaufen/d/das-haus-der-schwaene-a-j-cronin-geb-jubilaeumsausgabe-von-1934-208735636/;"
Annemarie";5;1230 Wien, 23. Bezirk, Liesing; ;16.28017;48.15508;Karl Schwed Gasse1230 Wien, 23. Bezirk, LiesingWien
Röhrenradio;http://www.willhaben.at/iad/kaufen-und-verkaufen/d/roehrenradio-208710345/;"

Can you post the csv, or a small amount that can be used to reproduce your results? — jacoblaw, Jun 23 '17 at 16:56
your reader is completely consumed by the `output2['lon'] = {row[6] for row in reader}` line. there is nothing left for lat. — stefan, Jun 23 '17 at 17:00
@ stefan, sorry it wasn't clear, but of course I took that in account. — fahrradlaus, Jun 23 '17 at 17:05

cs95 · Accepted Answer · 2017-06-23T17:25:48.000

2

Replace your set comprehension with a list comprehension. Also, your data is not consistent. To handle this, you should place your code in a try-except brace and also check for non-empty values.

import csv
with open('...', 'r') as f:
    reader = csv.reader(f, delimiter=';')
    output2 = { 'lat' : [], 'lon' : [] }
    for row in reader:
        try:
            if row[7] != '' and row[6] != '': 
                output2['lat'].append(row[7])
                output2['lon'].append(row[6])
        except:
            pass
print(output2)

Output:

{'lat': ['48.17209', '48.23059', '48.27574', '48.15508'], 'lon': ['16.35809', '16.37349', '16.35537', '16.28017']}

edited Jun 23 '17 at 17:25

answered Jun 23 '17 at 16:56

cs95

379,657
97
704
746

Apart from the fact that row[6] and row[7] are still right, see .csv file edit, this still gives me: `{'lat': set(['', '48.1803', '48.20549', ...], 'lon': set(['', '16.33052', '16.38002', ...])}` and not `{'lat': [...], 'lon': [...]}` – fahrradlaus Jun 23 '17 at 17:16
downvoted because of a careless answer. the idices are not off by one as as he gets logitude right. to make it worse you extend your answer with claiming an iterator beeing consumed twice. – stefan Jun 23 '17 at 17:17
I think your answer is ok, plus one for care in editing – developer_hatch Jun 23 '17 at 17:39
@DamianLattenero Thank you. I take my work seriously on this site. While the bad comments won't go away, I can at least do right by myself in not providing misinformation. – cs95 Jun 23 '17 at 17:45
@stefan I don't follow, what's wrong with Coldspeed answer? Exactly what line, and why? Because "your reader object is completely consumed" and "to make it worse you extend your answer with claiming an iterator beeing consumed twice" doensn't helps really... Don't take it wrong – developer_hatch Jun 23 '17 at 18:04
as a comment on the question (why there?) you wrote: *You lanced my answer for misinforming OP that the reader was consumed, when you said the same thing. Is this a joke? You should delete your comment*. Well you exactely wrote *your reader object is completely consumed **each** time* which is not true. it is consumed exactly once. You did not help the poster at all but led him in wrong directions for some time. Simply delete your answer and my comments are gone. Damian provided the correct answer anyway. – stefan Jun 23 '17 at 18:06
@Coldspeed Your original answer was flat wrong. It deserved a downvote and a comment as to why, which is what you got. You have since changed it significantly to be, deservedly, the accepted answer. Why all the arguing to muddy up your good answer with useless comments? – Alan Leuthard Jun 23 '17 at 18:45
@AlanLeuthard You're right. My immaturity shows sometimes. Will try to keep that in check as I go forward. Thanks for the wakeup call. :) – cs95 Jun 23 '17 at 18:48

score 2 · Answer 2 · edited Jun 23 '17 at 17:31

You're using set comprehension and attaching the set to a key. The problem with this is three fold:

1) You're stepping through your reader twice.

2) Sets are unordered.

3) Sets only store 1 copy of any value

Try this:

output2 = {'lat': [], 'lon': []}
with open('secondhand_data.csv', 'rb') as f:
    reader = csv.reader(f, delimiter=';')
    for row in reader:
        output2['lat'].append(row[7])
        output2['lon'].append(row[6])

developer_hatch · Answer 3 · 2017-06-23T18:06:00.953

2

You can do it simply like:

dct = {'lat': [], 'lon': []}
with open('secondhand_data.csv', 'rb') as f:
    reader = csv.reader(f, delimiter=';')
    for row in reader:
        dct['lat'] += list(row[7])
        dct['lon'] += list(row[6])

edited Jun 23 '17 at 18:06

answered Jun 23 '17 at 17:22

developer_hatch

15,898
3
42
75

Wouldn't this simply result in each key containing a list that contains the column values of only the final row? – Alan Leuthard Jun 23 '17 at 17:53
@AlanLeuthard yes you're right! I will edit right now, my bad haha – developer_hatch Jun 23 '17 at 17:59
So which is a more efficient way to add to a list? Using the + operator, or using append/extend? – Alan Leuthard Jun 23 '17 at 18:09
@AlanLeuthard I searched, append is twice faster, :) see here, https://stackoverflow.com/questions/725782/in-python-what-is-the-difference-between-append-and, I think now your answer is better haha, plus one for that – developer_hatch Jun 23 '17 at 18:12

Get dictionary from csv file by conversion

3 Answers3