6

I have a very large file, over 100GB (many billions of lines), and I would like to conduct a two-level sort as quick as possible on a unix system with limited memory. This will be one step in a large perl script, so I'd like to use perl if possible.

So, how can I do this? My data looks like this:

A    129
B    192
A    388
D    148
D    911
A    117

...But for billions of lines. I need to first sort by letter, and then by number. Would it be easier to use a unix sort, like...

sort -k1,2 myfile

Or can I do this all in perl somehow? My system will have something like 16GB ram, but the file is about 100GB.

Thanks for any suggestions!

Prix
  • 19,417
  • 15
  • 73
  • 132
jake9115
  • 3,964
  • 12
  • 49
  • 78
  • Are all the numbers 3 digits? If not, are they right-aligned? If both these conditions hold (all 3 digits or right-aligned) then you only need a single-level textual sort. – Jim Garrison Aug 12 '13 at 17:20
  • @Jim, thanks for the comment. No, the numbers range from 1-100,000,000 and these are just two, non-adjacent columns of a larger spreadsheet (genome sequencing data) – jake9115 Aug 12 '13 at 17:22

3 Answers3

8

The UNIX sort utility can handle sorting large data (e.g. larger than your working 16GB of RAM) by creating temporary working files on disk space.

So, I'd recommend simply using UNIX sort for this as you've suggested, invoking the option -T tmp_dir, and making sure that tmp_dir has enough disk space to hold all of the temporary working files that will be created there.

By the way, this is discussed in a previous SO question.

Community
  • 1
  • 1
asf107
  • 1,118
  • 7
  • 22
1

The UNIX sort is the best option for sorting data of this scale. I would recommend use fast compression algorithm LZO for it. It is usually distributed as lzop. Set big sort buffer using -S option. If you have some disk faster than then where you have default /tmp set also -T. Also, if you want sort by a number you have to define sorting number sorting as second sorting field. So you should use line like this for best performance:

LC_ALL=C sort -S 90% --compress-program=lzop -k1,1 -k2n
Hynek -Pichi- Vychodil
  • 26,174
  • 5
  • 52
  • 73
0

I had the exact same issue! After searching a lot and since i did not want any dependency on the shell (UNIX) to make it portable on windows i came up with the below solution:

#!/usr/bin/perl
use File::Sort qw(sort_file);
my $src_dic_name = 'C:\STORAGE\PERSONAL\PROJECTS\perl\test.txt';
sort_file({k => 1, t=>"    ", I => $src_dic_name, o => $src_dic_name.".sorted"});

I know this is a old post but updating it with the solution so that it is easy to find.

Documentation here

Prasad
  • 196
  • 9