I want to extract sentences that containing a drug and gene name from 10,000 articles

Question

I want to extract sentences that containing a drug and gene name from 10,000 articles. and my code is

import re
import glob
import fnmatch
import nltk
from nltk.tokenize import sent_tokenize, word_tokenize


flist= glob.glob ("C:/Users/Emma Belladona/Desktop/drug working/*.txt")
print (flist)
for txt in flist:
    #print (txt)
    fr = open (txt, "r")
    tmp = fr.read().strip()
    a = (sent_tokenize(tmp))
    b = (word_tokenize(tmp))
    for c, value in enumerate(a, 1):
        if value.find("SLC22A1") != -1 and value.find("Metformin"):
            print ("Result", value)
            re.findall("\w+\s?[gene]+", a)
        else:
            if value.find("Metformin") != -1 and value.find("SLC22A1"):
                print ("Results", value)
        if value.find("SLC29B2") != -1 and value.find("Metformin"):
            print ("Result", value)

I want to extract sentences that have gene and drug name from the whole body of article. For example "Metformin decreased logarithmically converted SLC22A1 excretion (from 1.5860.47 to 1.0060.52, p¼0.001)." "In conclusion, we could not demonstrate striking associations of the studied polymorphisms of SLC22A1, ACE, AGTR1, and ADD1 with antidiabetic responses to metformin in this well-controlled study."

This code return a lot of sentences i.e if one word of above came into the sentence that get printed out...! Help me making the code for this

Answer 1

You don't show your real code, but the code you have now has at least one mistake that would lead to lots of spurious output. It's on this line:

re.findall("\w+\s?[gene]+", a)

This regexp does not match strings containing gene, as you clearly intended. It matches (almost) any string contains one of the letters g, e or n.

This cannot be your real code, since a is a list and you would get an error on this line-- plus you ignore the results of the findall()! Sort out your question so it reflects reality. If your problem is still not solved, edit your question and include at least one sentence that is part of the output but you do NOT want to be seeing.

Answer 2

When you do this:

if value.find("SLC22A1") != -1 and value.find("Metformin"):

You're testing for "SLC22A1 in the string and "Metformin" not at the start of the string (the second part is probably not what you want)

You probably wanted this:

if value.find("SLC22A1") != -1 and value.find("Metformin") != -1:

This find method is error-prone due to its return value and you don't care for the position, so you'd be better off with in.

To test for 2 words in a sentence (possibly case-insensitive for the 2nd occurrence) do like this:

if "SLC22A1" in vlow and "metformin" in value.lower():

Answer 3

I'd take a different approach:

1. Read in the text file
2. Split the text file into sentences. Check out https://stackoverflow.com/a/28093215/223543 for a hand-rolled approach to do this. Or you could use the ntlk.tokenizer.punkt module. (Edited after Alexis pointed me in the right direction in the comments below).
3. Check if I find your key terms in each sentence and print if I do.

As long as your text files are well formatted, this should work.