1

I'm looking to achieve two things, reading large files which I've read up using mmap is suitable for this and reading the file in reverse (starting from the last line working up to the top).

I've tried both ways separately as follows:

f = open('syslog')
s = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)

#mmap method 
for line in iter(s.readline, ""):
    #search for string

#reading in reverse
for line in reversed(f.readlines()):
    #search for string

I'm struggling to incorporate both of these into the one and not sure of the best way to approach this.

Any help would be appreciated.

Cheers

Kev
  • 113
  • 1
  • 13
  • Uh, `f.readlines()` materializes your iterator into a list... Are you asking how to read through the memory-mapped file in reverse without materializing it into a list? – juanpa.arrivillaga Sep 05 '17 at 21:07
  • @CharlesDuffy it's definitely not a trivial task. I found a library that does it, though: https://pypi.python.org/pypi/file-read-backwards – juanpa.arrivillaga Sep 05 '17 at 21:11
  • Ahh; I wrote it as an answer on [read a file in reverse order using python](https://stackoverflow.com/questions/2301789/read-a-file-in-reverse-order-using-python), but then deleted it (presumably on finding a bug). There are some answers that use `seek()` efficiently, though. – Charles Duffy Sep 05 '17 at 21:11
  • Anyway, I think that if you want to simply iterate over the lines, `mmap` is not necessary... – juanpa.arrivillaga Sep 05 '17 at 21:15
  • The end goal is essentially to search for various strings in a dictionary and print the line if found. I'm open to alternative suggestions. Would a for loop work if using something along the lines of `if s.find(dict_key) != -1:` rather than iterating through each line. Not sure how I'd do it in reverse though – Kev Sep 05 '17 at 21:29
  • if you're only reading a large file, you shouldn't need to `mmap` it. Instead, open it normally and play games with `seek` to read it in reverse. See [this answer](https://stackoverflow.com/a/23646049/2958070) to easily do that. – Ben Sep 05 '17 at 21:30
  • What if you chunked the file (n lines) and then read each chunk in reverse? – pylang Sep 05 '17 at 23:49
  • a = buf[::-1] will cause MemoryError on big files, no idea why, it should load stuff into memory. – Vincent Alex May 10 '22 at 15:57

0 Answers0