5

I have a flat file (.txt) with 606,347 columns and I want to extract 50,000 RANDOM columns, with exception of the first column, which is sample identification. How can I do that using Linux commands? My file looks like:

ID  SNP1    SNP2    SNP3
1   0   0   2
2   1   0   2
3   2   0   1
4   1   1   2
5   2   1   0

It is TAB delimited.

Thank you so much.

Cheers,

Paula.

PaulaF
  • 393
  • 3
  • 17

3 Answers3

4

awk to the rescue!

$ cat shuffle.awk

   function shuffle(a,n,k) {
     for(i=1;i<=k;i++) {
       j=int(rand()*(n-i))+i
       if(j in a) a[i]=a[j]
       else a[i]=j
       a[j]=i;
     }
   }

   BEGIN {srand()}
   NR==1 {shuffle(ar,NF,ncols)}
         {for(i=1;i<=ncols;i++) printf "%s", $(ar[i]) FS; print ""}

general usage

$ echo $(seq 5) | awk -f shuffle.awk -v ncols=5
3 4 1 5 2

in your special case you can print $1 and start the function loop from 2.

i.e. change

for(i=1;i<=k;i++) to a[1]=1; for(i=2;i<=k;i++)

karakfa
  • 66,216
  • 7
  • 41
  • 56
  • Good answer. Check this for "better" randomness: http://stackoverflow.com/a/4048538/171318 – hek2mgl Mar 23 '16 at 21:14
  • yes, good point but I'm not sure it applies for this case since running for > 600k columns will take some time. Who comes with record structures like this? – karakfa Mar 23 '16 at 21:17
  • @karakfa thank you so much. That is working for me. The only problem is because I`m not being able to separate column 1 from the others. How do I ask to print it as first column and not include it in the random selection? Sorry, I`m a beginner on Unix commands. – PaulaF Mar 23 '16 at 21:20
  • 1
    @PaulaF, please see the update. I think the proposed change should be enough. – karakfa Mar 23 '16 at 21:31
2

Try this:

echo {2..606347} | tr ' ' '\n' | shuf | head -n 50000 | xargs -d '\n' | tr ' ' ',' | xargs -I {} cut -d $'\t' -f {} file

Update:

echo {2..606347} | tr ' ' '\n' | shuf | head -n 50000 | sed 's/.*/&p/' | sed -nf - <(tr '\t' '\n' <file) | tr '\n' '\t'
Cyrus
  • 84,225
  • 14
  • 89
  • 153
  • I get this error: xargs: argument list too long. Is there anything I can change? Thanks. :) – PaulaF Mar 23 '16 at 20:48
0

@karakfa 's answer is great, but the NF value can't be obtained in the BEGIN{} part of the awk script. Refer to: How to get number of fields in AWK prior to processing

I edited the code as: 

head -4 10X.txt | awk '
function shuffle(a,n,k){
  for(i=1;i<=k;i++) {
    j=int(rand()*(n-i))+i
    if(j in a) a[i]=a[j]
    else a[i]=j
    a[j]=i;
  }
}
BEGIN{
  FS=" ";OFS="\t"; ncols=10;
  }NR==1{shuffle(tmp_array,NF,ncols);
    for(i=1;i<=ncols;i++){
      printf "%s", $(tmp_array[i]) OFS;
    }
    print "";
  }NR>1{
    printf "%s", $1 OFS;
    for(i=1;i<=ncols;i++){    
      printf "%s", $(tmp_array[i]+1) OFS;
    }
    print "";
    }'

Because I am processing the single-cell gene expression profiles, so from the second row, the first column will be gene names. My output is:

D4-2_3095   D6-1_3010   D16-2i_1172 D4-1_337    iPSCs-2i_227    D4-2_170    D12-serum_1742  D4-1_1747   D10-2-2i_1373   D4-1_320    
Sox17   0   0   0   0   0   0   0   0   0   0   
Mrpl15  0.987862442831866   1.29176904082314    2.12650693025845    0   1.33257747910871    0   1.58815046312948    1.18541326956528    1.12103842107813    0.656789854017254   
Lypla1  0   1.29176904082314    0   0   0.443505832809852   0.780385141793088   0.57601629238987    0   0   0.656789854017254
Minstein
  • 542
  • 5
  • 10