Extracting multiple sequences from a 7 GB fasta file using IDs from a separate text file

Question

After searching this website for hours and trying many different things that didn't work, I have decided to post my own question. I currently have a text file (id.txt) that contains about 100 lines of the following IDS in this form:

5377-P3-D5-MSITS2a_R1reads1_1125821

5377-P3-D5-MSITS2a_R1reads1_1126992

I have a 7 GB fasta file with entries in the form

>5377-P3-D5-MSITS2a_R1reads1_1125821 M00532:203:000000000-BKM3D:1:1101:10654:16493 1:N:0:213 orig_bc=AAAAAAAAAAAA new_bc=AAAAAAAAAAAA bc_diffs=0    
AAGTCGTAACAAGGTCTCCGTAGGTGAACCTGCGGAGGGATCATTACACAAATATGAAGGCGGGCTGGAACCTCTCGGGGTTACAGCCTTGCTGAATTATTCACCCTTGTCTTTTGCGTACTTCTTGTTTCCTTGGTGGGTTCGCCCACCACTAGGACAAACATAAACCTTTTGTATTGGCA

>5377-P3-D5-MSITS2a_R1reads1_1126992 M00532:203:000000000-BKM3D:1:1104:27124:5463 1:N:0:213 orig_bc=AAAAAAAAAAAA new_bc=AAAAAAAAAAAA bc_diffs=0 
AAGTCGTAACAAGGTCTCCGTAGGTGAACCTGCGGAGGGATCATTACACAAATATGAAGGCGGGCTGGACCCTCTCGGGGTTACAGCCTTGCTGAATTATTCACCCTTGTCTTTTGCGTACATCTTGTTTCCTTTGTTGTTTCTCCCACCCCTAGGACAAACATAAACCTTTAGTAATTTCAATCAGCGT  

>5377-P3-D5-MSITS2a_R1reads1_1129826 M00532:203:000000000-BKM3D:1:1110:14480:9405 1:N:0:213 orig_bc=AAAAAAAAAAAA new_bc=AAAAAAAAAAAA bc_diffs=0 
AAGTCGTAACAAGGTCTCCGTAGGTGAACCTGCGGAGGGATCATTACACAAATATGAAGGCGGGCTGGAAACTCTCGAGGTTACAGCCTTGCTGAATTATTAACCCTTGTCGTTCGCGTACTTCTTGTTTCCTTGGTGTGTTCGCCCACCACAAGTAAAAACATAAACCTTTTGTAA

All IDs from id.text can be found in seq.fasta. The expected output would find the matching ID number in the fasta file from the id.text file and produce:

>5377-P3-D5-MSITS2a_R1reads1_1125821 M00532:203:000000000-BKM3D:1:1101:10654:16493 1:N:0:213 orig_bc=AAAAAAAAAAAA new_bc=AAAAAAAAAAAA bc_diffs=0    
AAGTCGTAACAAGGTCTCCGTAGGTGAACCTGCGGAGGGATCATTACACAAATATGAAGGCGGGCTGGAACCTCTCGGGGTTACAGCCTTGCTGAATTATTCACCCTTGTCTTTTGCGTACTTCTTGTTTCCTTGGTGGGTTCGCCCACCACTAGGACAAACATAAACCTTTTGTATTGGCA  

>5377-P3-D5-MSITS2a_R1reads1_1126992 M00532:203:000000000-BKM3D:1:1104:27124:5463 1:N:0:213 orig_bc=AAAAAAAAAAAA new_bc=AAAAAAAAAAAA bc_diffs=0 
AAGTCGTAACAAGGTCTCCGTAGGTGAACCTGCGGAGGGATCATTACACAAATATGAAGGCGGGCTGGACCCTCTCGGGGTTACAGCCTTGCTGAATTATTCACCCTTGTCTTTTGCGTACATCTTGTTTCCTTTGTTGTTTCTCCCACCCCTAGGACAAACATAAACCTTTAGTAATTTCAATCAGCGT

Currently, I am able to extract one sequence from the fasta file at a time using grep in bash by just copying and pasting one ID from the file.

Ex: grep 5377-P3-D5-MSITS2a_R1reads1_1126992 seq.fasta -A 1

Result:

>5377-P3-D5-MSITS2a_R1reads1_1126992 M00532:203:000000000-BKM3D:1:1104:27124:5463 1:N:0:213 orig_bc=AAAAAAAAAAAA new_bc=AAAAAAAAAAAA bc_diffs=0 AAGTCGTAACAAGGTCTCCGTAGGTGAACCTGCGGAGGGATCATTACACAAATATGAAGGCGGGCTGGACCCTCTCGGGGTTACAGCCTTGCTGAATTATTCACCCTTGTCTTTTGCGTACATCTTGTTTCCTTTGTTGTTTCTCCCACCCCTAGGACAAACATAAACCTTTAGTAATTTCAATCAGCGT

However, I have multiple text files, each containing anywhere from 50-300 IDs, that I would like to use to extract sequences from the FASTA file, and singularly extracting sequences seems unnecessarily time-consuming. I would like to find a way to find and output the sequences from the fasta file for multiple IDs located in a separate text file. I have mainly experimented with awk and grep commands in bash, based primarily on other answers on this site, and almost every command I try produces no result and no error message.

Examples I have tried:

awk -F '>' 'NR==FNR{ids[$0]; next} NF>1{f=($2 in ids)}f' id.txt seq.fasta

awk 'NR==FNR{ids[$0];next} /^>/{f=($1 in ids)} f' id.txt seq.fasta

grep -Fwf id.txt seq.fasta

grep -Ff id.txt seq.fasta

I feel like I have tried many variations of these two commands (based on other stack overflow and biostar suggestions) and in bash, nothing happens, no result or no error message. I am a relative beginner at coding as well, so I cannot pinpoint exactly what is going wrong. I am also open to any python or other code that could be used as well. Any help or advice would be appreciated. Thanks!

score 0 · Answer 1 · answered May 13 '20 at 15:32

The grep seems like the best idea to me. I think you might need to remove the * characters from your search strings though as they don't match what is in the file. With this change, it seems to work when I try your extract:

$ cat fasta 
*>5377-P3-D5-MSITS2a_R1reads1_1125821 M00532:203:000000000-BKM3D:1:1101:10654:16493 1:N:0:213 orig_bc=AAAAAAAAAAAA new_bc=AAAAAAAAAAAA bc_diffs=0   
AAGTCGTAACAAGGTCTCCGTAGGTGAACCTGCGGAGGGATCATTACACAAATATGAAGGCGGGCTGGAACCTCTCGGGGTTACAGCCTTGCTGAATTATTCACCCTTGTCTTTTGCGTACTTCTTGTTTCCTTGGTGGGTTCGCCCACCACTAGGACAAACATAAACCTTTTGTATTGGCA* 

*>5377-P3-D5-MSITS2a_R1reads1_1126992 M00532:203:000000000-BKM3D:1:1104:27124:5463 1:N:0:213 orig_bc=AAAAAAAAAAAA new_bc=AAAAAAAAAAAA bc_diffs=0    
AAGTCGTAACAAGGTCTCCGTAGGTGAACCTGCGGAGGGATCATTACACAAATATGAAGGCGGGCTGGACCCTCTCGGGGTTACAGCCTTGCTGAATTATTCACCCTTGTCTTTTGCGTACATCTTGTTTCCTTTGTTGTTTCTCCCACCCCTAGGACAAACATAAACCTTTAGTAATTTCAATCAGCGT* 

*>5377-P3-D5-MSITS2a_R1reads1_1129826 M00532:203:000000000-BKM3D:1:1110:14480:9405 1:N:0:213 orig_bc=AAAAAAAAAAAA new_bc=AAAAAAAAAAAA bc_diffs=0    
AAGTCGTAACAAGGTCTCCGTAGGTGAACCTGCGGAGGGATCATTACACAAATATGAAGGCGGGCTGGAAACTCTCGAGGTTACAGCCTTGCTGAATTATTAACCCTTGTCGTTCGCGTACTTCTTGTTTCCTTGGTGTGTTCGCCCACCACAAGTAAAAACATAAACCTTTTGTAA*
$ cat ids.txt 
5377-P3-D5-MSITS2a_R1reads1_1125821
5377-P3-D5-MSITS2a_R1reads1_1126992
$ grep -Ff ids.txt fasta 
*>5377-P3-D5-MSITS2a_R1reads1_1125821 M00532:203:000000000-BKM3D:1:1101:10654:16493 1:N:0:213 orig_bc=AAAAAAAAAAAA new_bc=AAAAAAAAAAAA bc_diffs=0   
*>5377-P3-D5-MSITS2a_R1reads1_1126992 M00532:203:000000000-BKM3D:1:1104:27124:5463 1:N:0:213 orig_bc=AAAAAAAAAAAA new_bc=AAAAAAAAAAAA bc_diffs=0 
$

Apologies, those asterisks were not there when I was trying different codes, and something formatted weirdly when posting. I am looking not just for the header output, but also the sequence output as well. However, even when I use the grep command with both files, I do not get any output at all. Is there any possibility my fasta file is too large? Thanks. — Emily W, May 13 '20 at 15:41
If you add the option `-A1` to the grep command, it should print the matched line plus one line after it. If you post your actual input and command, perhaps it will be clearer why you're not getting any output. As you can see, it works fine when I try it. — RGoodman, May 13 '20 at 15:48
If I create a new file with the few samples I listed in my question, it works fine for me as well. Still, there is no output at all when I use the full fasta file, which is why I am concerned. I may try splitting my fasta file up, to see if that helps. — Emily W, May 13 '20 at 15:55

score 0 · Answer 2 · answered May 28 '20 at 01:03

I found that dropping the -F flag and just using -f for the input file of patterns and -A 1 to also retrieve the sequence worked great, with the solution being:
grep -A 1 -f ids.txt seq.fasta

Also, if you don't want the -- separator between retrieved entries, add on a | grep -v "\-\-" to remove those lines (backslashes needed to escape the hyphens).

Full output:

% cat ids.txt
5377-P3-D5-MSITS2a_R1reads1_1125821
5377-P3-D5-MSITS2a_R1reads1_1126992

% cat seq.fasta
>5377-P3-D5-MSITS2a_R1reads1_1125821 M00532:203:000000000-BKM3D:1:1101:10654:16493 1:N:0:213 orig_bc=AAAAAAAAAAAA new_bc=AAAAAAAAAAAA bc_diffs=0
AAGTCGTAACAAGGTCTCCGTAGGTGAACCTGCGGAGGGATCATTACACAAATATGAAGGCGGGCTGGAACCTCTCGGGGTTACAGCCTTGCTGAATTATTCACCCTTGTCTTTTGCGTACTTCTTGTTTCCTTGGTGGGTTCGCCCACCACTAGGACAAACATAAACCTTTTGTATTGGCA

>5377-P3-D5-MSITS2a_R1reads1_1126992 M00532:203:000000000-BKM3D:1:1104:27124:5463 1:N:0:213 orig_bc=AAAAAAAAAAAA new_bc=AAAAAAAAAAAA bc_diffs=0
AAGTCGTAACAAGGTCTCCGTAGGTGAACCTGCGGAGGGATCATTACACAAATATGAAGGCGGGCTGGACCCTCTCGGGGTTACAGCCTTGCTGAATTATTCACCCTTGTCTTTTGCGTACATCTTGTTTCCTTTGTTGTTTCTCCCACCCCTAGGACAAACATAAACCTTTAGTAATTTCAATCAGCGT

>5377-P3-D5-MSITS2a_R1reads1_1129826 M00532:203:000000000-BKM3D:1:1110:14480:9405 1:N:0:213 orig_bc=AAAAAAAAAAAA new_bc=AAAAAAAAAAAA bc_diffs=0
AAGTCGTAACAAGGTCTCCGTAGGTGAACCTGCGGAGGGATCATTACACAAATATGAAGGCGGGCTGGAAACTCTCGAGGTTACAGCCTTGCTGAATTATTAACCCTTGTCGTTCGCGTACTTCTTGTTTCCTTGGTGTGTTCGCCCACCACAAGTAAAAACATAAACCTTTTGTAA

% grep -A 1 -f ids.txt seq.fasta
>5377-P3-D5-MSITS2a_R1reads1_1125821 M00532:203:000000000-BKM3D:1:1101:10654:16493 1:N:0:213 orig_bc=AAAAAAAAAAAA new_bc=AAAAAAAAAAAA bc_diffs=0
AAGTCGTAACAAGGTCTCCGTAGGTGAACCTGCGGAGGGATCATTACACAAATATGAAGGCGGGCTGGAACCTCTCGGGGTTACAGCCTTGCTGAATTATTCACCCTTGTCTTTTGCGTACTTCTTGTTTCCTTGGTGGGTTCGCCCACCACTAGGACAAACATAAACCTTTTGTATTGGCA
--
>5377-P3-D5-MSITS2a_R1reads1_1126992 M00532:203:000000000-BKM3D:1:1104:27124:5463 1:N:0:213 orig_bc=AAAAAAAAAAAA new_bc=AAAAAAAAAAAA bc_diffs=0
AAGTCGTAACAAGGTCTCCGTAGGTGAACCTGCGGAGGGATCATTACACAAATATGAAGGCGGGCTGGACCCTCTCGGGGTTACAGCCTTGCTGAATTATTCACCCTTGTCTTTTGCGTACATCTTGTTTCCTTTGTTGTTTCTCCCACCCCTAGGACAAACATAAACCTTTAGTAATTTCAATCAGCGT

% grep -A 1 -f ids.txt seq.fasta | grep -v "\-\-"
>5377-P3-D5-MSITS2a_R1reads1_1125821 M00532:203:000000000-BKM3D:1:1101:10654:16493 1:N:0:213 orig_bc=AAAAAAAAAAAA new_bc=AAAAAAAAAAAA bc_diffs=0
AAGTCGTAACAAGGTCTCCGTAGGTGAACCTGCGGAGGGATCATTACACAAATATGAAGGCGGGCTGGAACCTCTCGGGGTTACAGCCTTGCTGAATTATTCACCCTTGTCTTTTGCGTACTTCTTGTTTCCTTGGTGGGTTCGCCCACCACTAGGACAAACATAAACCTTTTGTATTGGCA
>5377-P3-D5-MSITS2a_R1reads1_1126992 M00532:203:000000000-BKM3D:1:1104:27124:5463 1:N:0:213 orig_bc=AAAAAAAAAAAA new_bc=AAAAAAAAAAAA bc_diffs=0
AAGTCGTAACAAGGTCTCCGTAGGTGAACCTGCGGAGGGATCATTACACAAATATGAAGGCGGGCTGGACCCTCTCGGGGTTACAGCCTTGCTGAATTATTCACCCTTGTCTTTTGCGTACATCTTGTTTCCTTTGTTGTTTCTCCCACCCCTAGGACAAACATAAACCTTTAGTAATTTCAATCAGCGT

Extracting multiple sequences from a 7 GB fasta file using IDs from a separate text file

2 Answers2