7

I have a pretty big file ~ 1MB in size and I would like to be able to read first N lines, save them into a list (newlist) for later use, and then delete them.

I am able to do that like this:

import os

n = 3 #the number of line to be read and deleted

with open("bigFile.txt") as f:
    mylist = f.read().splitlines()

newlist = mylist[:n]
os.remove("bigFile.txt")

thefile = open('bigFile.txt', 'w')

del mylist[:n]

for item in mylist:
  thefile.write("%s\n" % item)

I know this doesn't look good in terms of efficiency and this is why I need something better but after searching for different solutions I was stuck with this one.

Stephanie Safflower
  • 177
  • 2
  • 2
  • 9
  • you mean you want to remove the first lines of your file? – Jean-François Fabre Mar 06 '17 at 16:31
  • You are pretty much stuck. You can't remove from the beginning of a file without rewriting everything after it. – Mark Tolonen Mar 06 '17 at 16:32
  • @Chris_Rands not really a duplicate of that one. She knows how to delete the lines, just was hoping for a more efficient way. – Mark Tolonen Mar 06 '17 at 16:32
  • 1
    @MarkTolonen The 2nd answer in that question shows a more efficent way. – Aran-Fey Mar 06 '17 at 16:35
  • 1
    @MarkTolonen It's the same question, and I think it's fair to assume that the original question also wanted an efficient way too – Chris_Rands Mar 06 '17 at 16:38
  • 1
    I've marked as duplicate but posted an answer, as the current solutions look a bit weak to me. – Jean-François Fabre Mar 06 '17 at 16:39
  • Forgive my ignorance but where can I see the original question? Also, where did you post your answer? – Stephanie Safflower Mar 06 '17 at 16:54
  • follow the "duplicate" link on top of the page (pink background). Bring on the upvotes :) – Jean-François Fabre Mar 06 '17 at 16:57
  • Thank you for the heads up! I already have seen that question but the accepted answer wasn't what I was looking for as it created a new file. Your solution worked like a charm and I did upvoted but it seems I don't have sufficient privileges in order for my vote to show - "Thanks for the feedback! Votes cast by those with less than 15 reputation are recorded, but do not change the publicly displayed post score" – Stephanie Safflower Mar 06 '17 at 17:07
  • I just realized that I missed something - I need those first N values stored in a list just like I did in my example code. I missed this in my description even if I added this into my code. Does this makes my question original? – Stephanie Safflower Mar 06 '17 at 17:22

1 Answers1

9

A file is it's own iterator. 

n = 3
nfirstlines = []

with open("bigFile.txt") as f, open("bigfiletmp.txt", "w") as out:
    for x in xrange(n):
        nfirstlines.append(next(f))
    for line in f:
        out.write(line)

# NB : it seems that `os.rename()` complains on some systems
# if the destination file already exists.
os.remove("bigfile.txt")
os.rename("bigfiletmp.txt", "bigfile.txt")
bruno desthuilliers
  • 75,974
  • 6
  • 88
  • 118