PHP: sort a huge file on shared resources (limited to 1GB RAM)

Asked May 27 '17 at 21:23

Active May 27 '17 at 21:23

Viewed 146 times

I want to sort a really very huge file of approx 80GB, the file contains Lastnames on each line (separated by \r\n)

Because I am on shared resources with 1GB RAM available, is there a way to do this without putting all the data into a array in PHP?

I earlier came up with the idea to pick up chunks of 1GB, sort each chunk as separate sorted file, than I ended up with 80 sorted 1GB files on the file system. But I was quickly failed with this approach when I realized that I yet need to sort these 80 files with each other, for example reading two 1GB files will make 2048Megs which exceeds the size of my RAM so how would this work?

Considering the resources available to me, is this a possible task? Is there any other way?

asked May 27 '17 at 21:23

AamirR

11,672
4
59
73

1

Can't you use the OS sort functionality to sort the file? – Mark Baker May 27 '17 at 21:26
3

Or load it into a database, then rewrite via a SELECT with an ORDER BY clause – Mark Baker May 27 '17 at 21:27
Possible duplicate of [Sort a file with huge volume of data given memory constraint](https://stackoverflow.com/questions/2087469/sort-a-file-with-huge-volume-of-data-given-memory-constraint) – Traveling Tech Guy May 27 '17 at 21:31
@Mark use OS sort functionality using PHP, for example a unix linux shell sort utility, seems a good idea if this is possible, can you please show how to do it? – AamirR May 27 '17 at 21:33
[exec()](http://www.php.net/manual/en/function.exec.php) – Mark Baker May 27 '17 at 21:39
Why can't you run it through a database ? – Lucas Krupinski May 27 '17 at 22:17
@LucasKrupinski I do not have access and restricted not to use database – AamirR May 28 '17 at 17:58
Does your PHP have SQLite available? You can create a temporary database/table, and drop it again afterwards – Mark Baker May 29 '17 at 10:20

PHP: sort a huge file on shared resources (limited to 1GB RAM)

0 Answers0