How to efficiently get 10% of random lines out of the large file in Linux?

Question

I want to output random 10% lines of total lines of a file. For instance, file a has 1,000,000 lines then I want to output random 100,000 lines out of the file (100,000 being the 10% of 1,000,000) .

There is a easy to do this supposed that the file is small:

randomLine=`wc -l a | awk '{printf("%d\n",($1/10))}'`
sort -R a | head -n $randomLine

But using sort -R is very slow. It will perform a dedicated random computation. My file has 10,000,000 lines. Sorting takes too much time. Is there anyway to archive a less dedicated and not so random but efficient sampling?

Edit Ideas:

1. To sample a line every ten lines is acceptable. But I don't know how to do this with shell script.

2. Read line by line and if

   echo $RANDOM%100 | bc
   

is greater than 20 than output the line (Using the number greater than 10 to ensure get no less than 10% of line) and once output 10% line then stop. But I don't know how to read line by line using shell script.

Edit Description

The reason I want to use shell script is that my file contains \r characters. The new line character in the file should be \n but readline() function in Python and Java regards \r and \n as new line character, which doesn't fit my need.

Answer 1

Let's create a random list of X numbers from 1 to Y. You can do it with:

shuf -i 1-Y -nX

In your case,

shuf -i 1-1000000 -n10000

Then you store it in a variable (space separated) and pass to awk, so that you print those line numbers:

awk 'FNR==NR {a[$1]; next} {if (FNR in a) print}' <(shuf -i 1-1000000 -n10000) file

Explanation

• FNR==NR {a[$1]; next} loop through the shuf results and store them in a a[] array.
• {if (FNR in a) print} if the line number of the second parameter (the file) is found in the array a[], print it.

Sample with Y=10, X=2

$ cat a
1 hello
2 i am
3 fe
4 do
5 rqui
6 and
7 this
8 is 
9 sample
10 text

$ awk 'FNR==NR {a[$1]; next} {if (FNR in a) print}' <(shuf -i 1-10 -n2) a
2 i am
9 sample

$ awk 'FNR==NR {a[$1]; next} {if (FNR in a) print}' <(shuf -i 1-10 -n2) a
4 do
6 and

---

Improvement

As plundra suggested in comments:

shuf -n $(( $(wc -l < $FILENAME) / 10 )) $FILENAME

Answer 2

I think this is the best way:

file=your file here
lines_in_file=`wc -l < $file`
lines_wanted=$(($lines_in_file/10))

shuf -n $lines_wanted $file

Another creative solution:

echo $RANDOM generates a random number between 0 and 32767

Then, you can do:

echo $(($RANDOM*100000/32767+1))

.. to obtain a random number between 1 and 100000 (as nwellnhof points out in comments below, it's not any number from 1 to 100000, but one of 32768 possible numbers between 1 and 100000, so it's kind of a projection...)

So:

file=your file here
lines_in_file=`wc -l $file | awk {'print $1'}`
lines_wanted=$(($lines_in_file/10))
for i in `seq 1 $lines_wanted`
 do line_chosen=$(($RANDOM*${lines_in_file}/32767+1))
sed "${line_chosen}q;d" $file
done

Answer 3

I have this script that will give you roughly 1/x of the lines.

#!/usr/bin/perl -w

use strict;

my $ratio = shift;

while (<>) {
    print if ((rand) <= 1 / $ratio);
}

For a large enough $ratio, assuming a uniform distribution of rand's outputs.

Assuming you call this random_select_ratio.pl, run it like this to get 10% of the lines:

random_select_ratio.pl 10 my_file

or

cat my_file | random_select_ratio.pl 10

Answer 4

Just run this awk script with the file as input.

BEGIN { srand() }{ if (rand() < 0.10) print $0; }

It's been a while since I used awk, but I do believe that should do it.

And in fact it does work exactly as expected. Approximately 10% of the lines are output. On my Windows machine using GNU awk, I ran:

awk "BEGIN { srand() }{ if (rand() < 0.10) print $0; }" <numbers.txt >nums.txt

numbers.txt contained the numbers 1 through 1,000,000, one per line. Over multiple runs, the file nums.txt typically contained about 100,200 items, which works out to 10.02%.

If there's a problem with what awk considers a line, you can always change the record separator. That is RS = "\n"; But that should be the default on Linux machine.

Answer 5

Here's one way to do Edit idea 1. in bash:

while readarray -n10 a; do
    [ ${#a[@]} = 0 ] && break
    printf "%s" "${a[${RANDOM: -1:1}]}"
done < largefile.txt

Kinda slow, though it was about 2.5x faster than the sort -R method on my machine.

We use readarray to read from the input stream 10 lines at a time into an array. Then we use the last digit of $RANDOM as an index into that array and print the resulting line.

Using the readarray/printf combo should ensure the \r characters are passed through unmodified, as in the edited requirement.