Reading a binary file line by line in C++

Question

I'm a beginner in C++ so I hope you bear with me.

Trying to read a file which in text format each has lines that either look like this (the first few lines, called header lines):

@HD VN:1.5  SO:queryname

or like this

read.1  4   *   0   0   *   *   0   0   CAACCNNTACCACAGCCCGANGCATTAACAACTTAANNNCNNNTNNANNNNNNNNNNNNTTGAAAAAAAAAAAAAAAAAA    A<.AA##F..<F)<)FF))<#A<7<F.)FA.FAA.)###.###F##)############)FF)A<..A..7A....<F.A    XC:Z:CAACCNNTACCA   RG:Z:A  XQ:i:2

Both are tab delimited.

The file is very large and therefore is in binary format. I'm wondering whether it is possible to read from the binary format file each line, do some processing on that line, and then write it to a binary format output file.

I started with this code:

#include <iostream>
#include <fstream>
#include <string>
using namespace std;
int main(int argc, char* argv[])
{
  string input_file = argv[1];
  string output_file = argv[2];
  string line;
  ifstream istream;
  istream.open(input_file.c_str(),ios::binary|ios::in);
  ofstream ostream;
  ostream.open(output_file.c_str(),ios::binary|ios::out);
  while(getline(istream,line,'\n')){
    if(line.empty()) continue;
    //process line assuming it is read as a string
    ostream<<line<<endl;
  }
  istream.close();
  ostream.close();
}

But it crashes with: Segmentation fault (core dumped), in the part where I'm trying to parse line to a string vector.

Is there a way to read the binary format and split it by lines, do string processing on each such line, and then write them to a binary output?

BTW, I'm running this on Linux.

Answer 1

Is it possible to read a binary file line by line?

Every file is, in principle, binary, because that's just how computers work. Now, saying "I'm trying to read it line-by-line" clearly means you're treating it as a text file – "line" is a text concept.

The file is very large and therefore is in binary format.

That's top-notch bullshit. Size doesn't change the format of your file.

How do I get each line as a string? Does the ostream<<line<<endl; work for writing a string to a binary file?

Yes and no: if you're file is not a text file, why is it important where these '\n' characters are? To a non-text file, these are just normal bytes like 'a' or \0x00 or 0xFF. So basically, you're looking at and try to spot letters in there.

However, with your illustration of the files we're talking about, they are in fact files that only contain text.

So your problem seems to lie in the fact that a single line might exceed what storage you have available in std::string. That's a rare case – but it can happen for genetic strings, it seems. Well.

Get yourself familiar with the non-text-oriented file I/O that C++ has. Basically, there's ifstream.read() and you should use it to get a (limited) amount of bytes, do your processing, write to output, repeat. Look out for the newline character in your input, and "rewind" your file (fseek) if you've read past it.

Also, I really wonder how long your lines have to become to break std::string. I guess you might be running on some very limited OS (32 bit?) or computer (very little RAM + Swap?).

Answer 2

If your file is structured into lines, and each line is terminated with a \n then it is a text file. Every file is binary underneath, and text files are just a special kind of binary file.

So, given that, the code you've shown is likely to work fine for files of any size.

You should really remove the ios:binary, but I don't expect it to make any difference in this case.

But if you're getting a crash while "processing" a line of the file, that's where the the bug is most likely to be - in the code you haven't disclosed - yet!

Answer 3

It looks like you file has some other line endings than you expect. It could have a \r while you expect it to have \n. If that is the case then std::getline tries to read whole 30GB file in to line std::string.

I suggest you check what line ending you have in your file, to verify above. If that is the case then you can use line reading function from this SO: Getting std :: ifstream to handle LF, CR, and CRLF? which should read lines even if they have ending non compatible with your platform (or rather endings which you do not expect).

also, you should be fine using non-binary file mode. The sample file lines you have shown in question does not look very binary to me.