1

I am dealing with big text files that at some point won't fit in to my memory. My idea was to read them line by line (because you don't have to load the whole file in memory) so I won't have memory issues.

with open(file) as infile:
    for line in infile:
        # do stuff with this line

The issue comes because the way I create those files is with loading dictionaries in them:

def save_dictionary_to_txt_file(dic, file_name):
    import json
    with open(file_name, 'w') as fp:
        fp.write(json.dumps(dic))

So let's say that the dictionary I have is this one:

{'apple': ['5', '65'], 'blue': ['9', '10', '15', '43'], 'candle': ['15'], 'is': ['5', '6', '13', '45', '96']}

Then in the text file I have it in the exact same format as a line. Is there any way in which I could insert new lines in the text file without needing to parse the file again? or most important loading it in memory? I tried with pickle since it is a object serialiation module but didn't have any success. What I tried was to have like the '\n' new line character inside my dictionary every couple of keys.

{'apple': ['5', '65'], 'blue': ['9', '10', '15', '43'], '\n', 'candle': ['15'], 'is': ['5', '6', '13', '45', '96']}

I have tried with json files and it works (keeping the format of the dictionary) but json files have other issues that prevent me from using them in my algorithm. (for example you can't append another dictionary to a json file, or you can if you can load the whole file in memory which I can't do).

Gerasimos Ragavanis
  • 361
  • 1
  • 3
  • 16
  • 1
    Possible duplicate of [How to prettyprint a JSON file?](https://stackoverflow.com/questions/12943819/how-to-prettyprint-a-json-file) – meowgoesthedog Apr 23 '19 at 14:40
  • 2
    `json.dump(fp, dic, indent=4)` should do the trick. if not, `fp.write(json.dumps(dic, indent=4))` – Torxed Apr 23 '19 at 14:40
  • @meowgoesthedog it wasn't what I was asking for I think. – Gerasimos Ragavanis Apr 23 '19 at 14:44
  • 1
    Dumping to and loading from json isn't going to protect you from memory errors. Have you considered a database. Python comes with SQLite built in. – brunns Apr 23 '19 at 14:44
  • @Torxed thank you. I don't know why I didn't think about this. Kinda close minded there. It is a format that I could work with, not what I originally had in mind – Gerasimos Ragavanis Apr 23 '19 at 14:45
  • If you actually read the link, I believe you'll find that it is *exactly* what you were looking for. – meowgoesthedog Apr 23 '19 at 14:46
  • @meowgoesthedog but I am working with text files. Anyway I will check it, but user Torxed already gave me another solution I could work with. Appreciate your comment – Gerasimos Ragavanis Apr 23 '19 at 14:48

0 Answers0