Is there an elegant way to parse a text file *backwards*?

Question

Possible Duplicate:
How to read a file from bottom to top in Ruby?

In the course of working on my Ruby program, I had the Eureka Moment that it would be much simpler to write if I were able to parse the text files backwards, rather than forward.

It seems like it would be simple to simply read the text file, line by line, into an array, then write the lines backwards into a text file, parse this temp file forwards (which would now effectively be going backwards) make any necessary changes, re-catalog the resulting lines into an array, and write them backwards a second time, restoring the original direction, before saving the modifications as a new file.

While feasible in theory, I see several problems with it in practice, the biggest of which is that if the size of the text file is very large, a single array will not be able to hold the entirety of the document at once.

Is there a more elegant way to accomplish reading a text file backwards?

Answer 1

If you are not using lots of UTF-8 characters you can use Elif library which work just like File.open. just load Elif and replace File.open with Elif.open

Elif.open('read.txt', "r").each_line{ |s|
    puts s
}

This is a great library, but the only problem I am experiencing right now is that it have several problems with line ending in UTF-8. I now have to re-think a way to iterate my files

---

Additional Details

As I google a way to answer this problem for UTF-8 reverse file reading. I found a way that already implemented by File library:

To read a file backward you can try the ff code:

File.readlines('manga_search.test.txt').reverse_each{ |s|
   puts s
}

This can do a good job as well

Answer 2

There's no software limit to Ruby array. There are some memory limitations though: Array size too big - ruby

Your approach would work much faster if you can read everything into memory, operate there and write it back to disk. Assuming the file fits in memory of course.

Answer 3

Let's say your lines are 80 chars wide on average, and you want to read 100 lines. If you want it efficient (as opposed to implemented with the least amount of code), then go back 80*100 bytes from the end (using seek with the "relative to end" option), then read ONE line (this is likely a partial one, so throw it away). Remember your current position via tell, then read everything up until the end.

You now have either more or less than a 100 lines in memory. If less, go back (100+1.5*no_of_missing_lines)*80, and repeat the above steps, but only reading lines until you reach your remembered position from before. Rinse and repeat.

Answer 4

How about just going to the end of the file and iterating backwards over each char until you reach a newline, read the line, and so on? Not elegant, but certainly effective.

Example: https://gist.github.com/1117141

Answer 5

I can't think of an elegant way to do something so unusual as this, but you could probably do it using the file-tail library. It uses random access files in Ruby to read it backwards (and you could even do it yourself, look for random access at this link).

Answer 6

You could go throughout the file once forward, storing only the byte offset of each \n instead of storing the full string for each line. Then you traverse your offset array backward and can use ios.sysseek and ios.sysread to get lines out of the file. Unless your file is truly enormous, that should alleviate the memory issue.

Admitedly, this absolutely fails the elegance test.