How should I go about reading a large binary file

Question

I have a large binary file (more than 500mb) that I want to read a part of and extract data from. I'm sure I shouldn't load the whole file into memory all at once, so how could I load only a part of it, using hex offsets?

I've never worked with this kind of thing before, so I have no idea where to start. All I want to read and write to the file is text.

If you know the offset (the human representation of this offset, be it octal, decimal, or hex is totally irrelevant) then just seek to that part of the file and read the correct number of bytes. If this doesn't make sense to you, it's time for [a new book](http://stackoverflow.com/questions/388242/the-definitive-c-book-guide-and-list). — Lightness Races in Orbit, Jan 14 '13 at 21:06
@LightnessRacesinOrbit: He is working with text, so seeking may not be an option... — K-ballo, Jan 14 '13 at 21:08
@Sean: It's a binary file, but all you want to read and write is text? Huh? — Lightness Races in Orbit, Jan 14 '13 at 21:10
@LightnessRacesinOrbit Sorry for the confusion. I meant I wanted to write hex that in the end is represented as text. You, ultimately I want to take the hex in the file, convert it to a string, then display it in a list. — Sean Heiss, Jan 14 '13 at 21:12
@Kerrek: They're _prettier_! And can take you so much further... — Lightness Races in Orbit, Jan 14 '13 at 21:15
@KerrekSB Yes, as I'm using a hex editor, it's just more simple to use hex offsets. — Sean Heiss, Jan 14 '13 at 21:15
I understand HEX better than Decimal actually. Helps me do the math fast. — Aniket Inge, Jan 14 '13 at 21:16
@Sean: To decode the banter, there is no such thing as a "hex offset". There is just an offset, which is a number, which you can write in decimal or hex or whatever. If you want to write them in hex that's fine, but it doesn't make them "hex offsets". :) — Lightness Races in Orbit, Jan 14 '13 at 21:16
@Aniket I'm making a program to use with translation a Playstation 1 ISO. I have a text table that converts the "encrypted" hex into text, I just need to make a program so that a user can edit the file. — Sean Heiss, Jan 14 '13 at 21:17
@LightnessRacesinOrbit no, primary school was facinating. High school was MORE facinating :P — Aniket Inge, Jan 14 '13 at 21:19
So you're basically making a hex editor. Memory mapped file is how you go. :) @SeanHeiss — Aniket Inge, Jan 14 '13 at 21:20

score 6 · Accepted Answer · edited Jan 14 '13 at 21:46

6

Use a Memory Mapped File. This will allow you to process the file as if it were in memory, but it only ever gets loaded into memory a page at a time.

edited Jan 14 '13 at 21:46

Alex Chamberlain

4,147
2
22
49

answered Jan 14 '13 at 21:17

Otávio Décio

73,752
17
161
228

3

might also want to link [``mmap``](http://linux.die.net/man/2/mmap) for posix OSs, but the idea is just right :) – Jonas Schäfer Jan 14 '13 at 21:18
1

It does get loaded into memory, just one page at a time. – Alex Chamberlain Jan 14 '13 at 21:30
Great! That sounds perfect. Do you know of a guide or any other resource for using these in C++? – Sean Heiss Jan 14 '13 at 21:32
@Alex, correct - but it never gets totally loaded to memory. Thanks – Otávio Décio Jan 14 '13 at 21:33
@OtávioDécio That's for C#/VB. What about for C++? – Sean Heiss Jan 14 '13 at 21:36
http://www.codeproject.com/Articles/37201/Simple-File-I-O-Using-Windows-Memory-Mapped-Files – Alex Chamberlain Jan 14 '13 at 21:38

How should I go about reading a large binary file

1 Answers1