cat, sort, gzip of 20GB files

Asked Apr 01 '15 at 00:04

Active Apr 01 '15 at 00:04

Viewed 77 times

I am out of memory to perform all steps individually, therefore I was thinking or doing something like this:

cat r_T_H_17.bed r_T_H_stim.bed r_T_mem.bed r_T_naive.bed r_T_reg.bed | sort -k1,1 -k2,2n -k3,3n | bgzip > all_r.bed.gz

all files are min 12 and max 23 GB, I am intended to run it on a cluster. Does it mean I will have to sum sizes of all files to estimate the memory consumption?

asked Apr 01 '15 at 00:04

Alina

2,191
3
33
68

How is this doing individually ? `cat file1 file2 file3` would read all the files in memory. – Jagannath Apr 01 '15 at 00:16
`cat`, `sort` and `zip` can all operate on large files without exhausting memory. – StilesCrisis Apr 01 '15 at 00:16

cat, sort, gzip of 20GB files

0 Answers0