Building indexes for files in Perl

Question

I'm currently new to Perl, and I've stumbled upon a problem :

My task is to create a simple way to access a line of a big file in Perl, the fastest way possible. I created a file consisting of 5 million lines with, on each line, the number of the line. I've then created my main program that will need to be able to print any content of a given line. To do this, I'm using two methods I've found on the internet :

use Config qw( %Config );

my $off_t = $Config{lseeksize} > $Config{ivsize} ? 'F' : 'j';
my $file = "testfile.err";
open(FILE, "< $file")         or die "Can't open $file for reading: $!\n";
open(INDEX, "+>$file.idx")
        or die "Can't open $file.idx for read/write: $!\n";
build_index(*FILE, *INDEX);
my $line = line_with_index(*FILE, *INDEX, 129);
print "$line";

sub build_index {
    my $data_file  = shift;
    my $index_file = shift;
    my $offset     = 0;

    while (<$data_file>) {
        print $index_file pack($off_t, $offset);
        $offset = tell($data_file);
    }
}

sub line_with_index {
    my $data_file   = shift;
    my $index_file  = shift;
    my $line_number = shift;

    my $size;               # size of an index entry
    my $i_offset;           # offset into the index of the entry
    my $entry;              # index entry
    my $d_offset;           # offset into the data file

    $size = length(pack($off_t, 0));
    $i_offset = $size * ($line_number-1);
    seek($index_file, $i_offset, 0) or return;
    read($index_file, $entry, $size);
    $d_offset = unpack($off_t, $entry);
    seek($data_file, $d_offset, 0);
    return scalar(<$data_file>);
}

Those methods sometimes work, I get a value once out of ten tries on different set of values, but most of the time I get "Used of uninitialized value $line in string at test2.pl line 10" (when looking for line 566 in the file) or not the right numeric value. Moreover, the indexing seems to work fine on the first two hundred or so lines, but afterwards I get the error. I really don't know what I'm doing wrong..

I know you can use a basic loop that will parse each line, but I really need a way of accessing, at any given time, one line of a file without reparsing it all over again.

Edit : I've tried using a little tip found here : Reading a particular line by line number in a very large file I've replaced the "N" template for pack with :

my $off_t = $Config{lseeksize} > $Config{ivsize} ? 'F' : 'j';

It makes the process work better, until line 128, where instead of getting 128 , I get a blank string. For 129, I get 3, which doesn't mean much..

Edit2 : Basically what I need is a mechanism that enables me to read the next 2 lines for instance for a file that is already being read, while keeping the read "head" at the current line (and not 2 lines after).

Thanks for your help !

Answer 1

Since you are writing binary data to the index file, you need to set the filehandle to binary mode, especially if you are in Windows:

open(INDEX, "+>$file.idx")
    or die "Can't open $file.idx for read/write: $!\n";
binmode(INDEX);

Right now, when you perform something like this in Windows:

print $index_file pack("j", $offset);

Perl will convert any 0x0a's in the packed string to 0x0d0a's. Setting the filehandle to binmode will make sure line feeds are not converted to carriage return-line feeds.