1

I have a 100GB text-file i need to import to a mysql-db. So far I planned to use python to read the file with readline() and to delete imported lines in case the program crashes. Is there any way to delete the lines without loading the whole file into memory? Or is there a better approach to that task?

knpfl
  • 91
  • 2
  • There's no way to delete data at the start of the file. You could perhaps overwrite already-processed lines with an equal number of spaces or null characters, or you could save the number of processed lines in a separate file (and skip that many lines if your program is restarted). – jasonharper Jun 25 '21 at 14:55
  • I believe the `with open as` ctx manager already loads it as a file stream. Therefore only the line currently iterated is actually loaded into memory. – TheLazyScripter Jun 25 '21 at 14:56
  • @TheLazyScripter it's nothing to do with context managers. – Peter Wood Jun 25 '21 at 14:56

1 Answers1

0

Keep the line number of the last successfully completed query. If your script crashes, just start again from that line.

Simon Brahan
  • 2,016
  • 1
  • 14
  • 22
  • 1
    I don't get the downvote; seems like a legitimate approach. I'd look at perhaps using `n = fp.tell()` then `fp.seek(n)` though instead of counting lines. –  Jun 25 '21 at 15:06