0

I have received a text file generated by a json dump in python that looks like this:

[0.1,0.1,0.2,0.3]
[0.1,0.3,0.4,0.3]
[0.1,0.1,0.3,0.3]
[0.3,0.1,0.5,0.3]
.
.
.
[0.1,0.1,0.3,0.3]
[0.3,0.4,0.6,0.3]

and so on for a considerable amount of lines ~>10,000,000

I would like to figure out the quickest/most efficient way to read from the file and actually convert them into lists.

I have a program that has a for loop that runs a particular operation with lists:

for x in range(filelength):
    for y in list(each line from the file):
        use the numbers from each list to perform certain operations

I was thinking of parsing out all the brackets from the text file and feeding each value comma separated into a blank list for each line (which would probably be slow and time consuming), but I thought there might be a feature of python to convert a list represented as a string easily into an actual list in python quickly.

Any thoughts or suggestions would be appreciated.

Charles Jin
  • 3
  • 3
  • possible duplicate of [Loading & Parsing JSON file in python](http://stackoverflow.com/questions/12451431/loading-parsing-json-file-in-python) – Martijn Pieters Sep 17 '13 at 21:36

2 Answers2

6

Use ast.literal_eval() to parse each line back into a Python list:

import ast

with open(filename, 'r') as fh:
    for line in fh:
        listobj = ast.literal_eval(line)

ast.literal_eval() takes a string and interprets it as Python literal values; lists and floating point values are directly supported:

>>> ast.literal_eval('[0.1,0.1,0.2,0.3]\n')
[0.1, 0.1, 0.2, 0.3]
Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • But if he got this as JSON dumps, why evaluate it as Python literals? – abarnert Sep 17 '13 at 21:35
  • @abarnert: because I missed that part; then it is a dupe of another answer of mine. :-) – Martijn Pieters Sep 17 '13 at 21:36
  • Not surprising; 30% of the questions here, or 90% of the useful/answerable ones, are a dup of another answer of yours. :) – abarnert Sep 17 '13 at 21:55
  • Thanks! ast.literal_eval() was exactly what I needed. I have never used JSON before - I got my data from someone else who used it (and told me they used it.) I'm going to look into both the JSON encoder/decoder now along with ast so I know what to do in the future. – Charles Jin Sep 17 '13 at 22:33
1

You say this was "generated by a json dump", and each line looks like valid JSON, so the right thing to do is to parse each line as JSON:

import json
with open(filename) as f:
    the_lists = map(json.loads, f)

Since you just want to iterate directly over the lists, it might be simpler to do the loads right in your loop:

import json
with open(filename) as f:
    for line in f:
        for column in json.loads(line):
            # your code here
abarnert
  • 354,177
  • 51
  • 601
  • 671