0

I just started out running some bioinformatic scripts for variant calling from a friend of mine and I am still very new to this. Unfortunately, I get hard stuck when I get to the part of the script where I have to use GetBaseCountsMultiSample (https://github.com/zengzheng123/GetBaseCountsMultiSample) to calculate the base counts in the tumor and normal BAM file for all the sites in my VCF file. Every time I use the tool it terminates after loading the second BAM file and the output in the terminal is "killed".

I used WSL2 with Ubuntu 22.04 on Windows 10 with 80GB RAM and my command is:

cd ~/Output_Strelka
~/packages/GetBaseCountsMultiSample-1.2.3/GetBaseCountsMultiSample \
--fasta ~/Output_Strelka/MoCaSeq_ref/GRCh38.p12.reduced.fa --maq 1 \
--bam Tumor:M15/results/bam/M15RNA.Tumor.bam \
--bam Normal:M15/results/bam/M15.Normal.bam \
--vcf M15/results/rescued/M15.StrelkaRNA.vcf \
--output M15/results/rescued/M15.StrelkaRNA.M15.StrelkaRNA.vcf \
--thread 1

This is the output in the terminal:

[INFO] Loading reference sequence: /home/chiv/Output_Strelka/MoCaSeq_ref/GRCh38.p12.reduced.fa
[INFO] Finished loading reference sequence
[INFO] Loading variants file: M15/results/rescued/M15.StrelkaRNA.vcf
[INFO] 347217 variants has been loaded from file: M15/results/rescued/M15.StrelkaRNA.vcf
[INFO] Sorting variants
[INFO] Indexing variants
[INFO] Processing bam file: M15/results/bam/M15.Normal.bam
[INFO] Processing bam file: M15/results/bam/M15RNA.Tumor.bam
Killed

Here are some of the error messages that I retrieved from the log file:

/var/log/kern.log:Jul 30 20:33:43 DESKTOP-9R7FHMK kernel: [34890.795897] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=GetBaseCountsMu,pid=100281,uid=1000
/var/log/kern.log:Jul 30 20:33:43 DESKTOP-9R7FHMK kernel: [34890.795920] Out of memory: Killed process 100281 (GetBaseCountsMu) total-vm:49172696kB, anon-rss:39340272kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:95440kB oom_score_adj:0
/var/log/kern.log:Jul 30 20:36:38 DESKTOP-9R7FHMK kernel: [35066.182716] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=GetBaseCountsMu,pid=100477,uid=1000
/var/log/kern.log:Jul 30 20:36:38 DESKTOP-9R7FHMK kernel: [35066.182726] Out of memory: Killed process 100477 (GetBaseCountsMu) total-vm:49466188kB, anon-rss:39645536kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:95896kB oom_score_adj:0
/var/log/kern.log:Jul 30 21:15:20 DESKTOP-9R7FHMK kernel: [37388.345857] GetBaseCountsMu invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
/var/log/kern.log:Jul 30 21:15:20 DESKTOP-9R7FHMK kernel: [37388.345877]  oom_kill_process.cold+0xb/0x10
/var/log/kern.log:Jul 30 21:17:17 DESKTOP-9R7FHMK kernel: [37505.417357] GetBaseCountsMu invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
/var/log/kern.log:Jul 30 21:17:17 DESKTOP-9R7FHMK kernel: [37505.417377]  oom_kill_process.cold+0xb/0x10

After loading the second BAM file my ram goes from 20gb to 73gb usage in a matter of 30s, remains there for 3 minutes and then the task gets killed. CPU usage is about 25% the entire time.

I asked my friend that I got the script from but his suggestions did not solve the problem. He tried all my input files on his machine to rule out that this is the cause and on his system it works. Here is what I tried to do to fix the problem:

  1. I tried to run the task in a docker container from my friend but same problem
  2. I tried with Ubuntu 18.04 and even WSL1 as well as different g++ compilers to install the package
  3. I used the --max_block_size and --max_block_dist options of GetBaseCountsMultiSample
  4. I tried to disable OOM Kill and use sudo sysctl vm.overcommit_memory=2, but with the second option it stops with the error "terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc Aborted"
  5. I installed a native Ubuntu 22.04 LTS in dual boot with the windows boot manager to rule out that the problem comes from WSL but same error
  6. In a hail merry I removed my 16GB RAM Kit to have only my 64 GB Kit with two sticks but same issue
  7. I ran the same code on another computer at work with WSL2 Ubuntu 22.04 and 64GB Ram but same error
  8. I also used the newest version of GetBaseCountsMultiSample (and not just the version my friend uses) but to no avail

What somewhat worked: I reduced the file size of my BAM files with:

samtools view -h M15.Normal.bam | head -n 1000 | samtools view -bS - > M15.Normal.small.bam
samtools view -h M15RNA.Tumor.bam | head -n 1000 | samtools view -bS - > M15RNA.Tumor.small.bam

--> this actually worked without issue with GetBaseCountsMultiSample

I then fractioned my BAM files into individual chromosomes. Chromosome 1 worked (size 230MB) but Chromosome 2 (size 408 MB) already gave the OOM Kill error. Even fractioning Chr2 further in files below 200MB did not help.

As I said earlier, my input files work on my friends linux system with GetBaseCountsMultiSample. As far as I understand, dual booting linux should solve the problem if it is windows-related or could that still be an issue. I am extremely grateful for any advice.

Thanks everyone for their time and apologies for the long post. I wanted to give as much information about the error and my attempts.

Chiv
  • 1
  • 2
  • Hi Chiv. How much memory does you friends pipeline usually consume? (i.e. we have machines 1+ TB RAM too, and in bioinformatics we sometimes just have to deal with these numbers) How did you install this tool? It looks like you have to compile it yourself, did you do so or just copied binaries from elsewhere? Have you considered writing an issue in github directly? Maybe the original authors can help you. – Patrick H. Aug 08 '23 at 12:31
  • Hi Patrick, thank you so much for your advice. My friend`s system uses 256GB Ram. He can run the complete BAM files with 5GB size without any problem. My system with 80GB even crushes with files that are split into BAM files with 400MB size. So it could potentially be the RAM amount differences but my system should be able to run the smaller BAM files. I have no problems to perform alignment or variant calling on full sized BAM files with my system. – Chiv Aug 08 '23 at 19:16
  • For the installation I tried two ways: 1. I followed the instructions on the github page of the package and compiled the make file (to compile the make file I needed to install the old g++ version 4.8 because newer versions do not work). For the second way I compiled the included package bamtools separately and modified the make file (this is also suggested in known issues on github for the package and is how my friend installed it). Thanks for the advice, I will reach out to the original author on github but as far as I saw it does not receive updates anymore so my chances might be low. – Chiv Aug 08 '23 at 19:29
  • I also tried to use a swap file on my native Ubuntu 22.04 system. I created a 128gb swap file and confirmed with ´free -m´ that it is active. This again did not work with BAM files of 400MB size. Thank you for your help. It is very appreciated. – Chiv Aug 08 '23 at 19:32
  • I believe that this tool just takes a bit more memory than you have available on your machine. Not sure how you can subset it, maybe you can process reference and vcf-files chromosome-wise? It won't help to just create swap files, because it's just a single process that requires this much memory. – Patrick H. Aug 09 '23 at 13:23
  • Thank you, you were correct. It uses more memory than my machine has. However, creating a swap file with 300GB+ size was sufficient to run the tool. It uses all my 80GB ram + 200GB swap space. This is independent on the size of the input file (full-sized bams and reference or bams and reference split by chromosome). Probably just not a very hardware efficient tool but the workaround with the swap file at least made it usable. Thanks again! – Chiv Aug 09 '23 at 20:16

1 Answers1

0

Thank you Patrick H. for your helpful comments. I managed to fix it now with the advice from my friend:

I created a swap file with 300+ GB and monitored the ram consumption during usage. Apparently the program uses a ridiculous amount of ram independent of the size of the input files. When it ramps up it uses all my 80GB of installed ram and 200GB of my swap file (this is the same for small bam files and full-sized bam files). But with this it manages to complete the task.

Thanks again and I hope that helps others who run into similar issues.

Chiv
  • 1
  • 2