Use grep on file in Python

Question

I have searched the grep answers on here and cannot find an answer. They all seem to search for a string in a file, not a list of strings from a file. I already have a search function that works, but grep does it WAY faster. I have a list of strings in a file sn.txt (with one string on each line, no deliminators). I want to search another file (Merge_EXP.exp) for lines that have a match and write it out to a new file. The file I am searching in has a half millions lines, so searching for a few thousand in there takes hours without grep.

When I run it from command prompt in windows, it does it in minutes:

grep --file=sn.txt Merge_EXP.exp > Merge_EXP_Out.exp

How can I call this same process from Python? I don't really want alternatives in Python because I already have one that works but takes a while. Unless you think you can significantly improve the performance of that:

def match_SN(serialnumb, Exp_Merge, output_exp):
    fout = open(output_exp,'a')
    f = open(Exp_Merge,'r')
    # skip first line
    f.readline()
    for record in f:
        record = record.strip().rstrip('\n')
        if serialnumb in record:
            fout.write (record + '\n')
    f.close()
    fout.close()

def main(Output_CSV, Exp_Merge, updated_exp):

    # create a blank output
    fout = open(updated_exp,'w')

    # copy header records
    f = open(Exp_Merge,'r')
    header1 = f.readline()
    fout.write(header1)
    header2 = f.readline()
    fout.write(header2)
    fout.close()
    f.close()

    f_csv = open(Output_CSV,'r')
    f_csv.readline()
    for rec in f_csv:
        rec_list = rec.split(",")
        sn = rec_list[2]
        sn = sn.strip().rstrip('\n')
        match_SN(sn,Exp_Merge,updated_exp)

os.system("grep --file=sn.txt Merge_EXP.exp > Merge_EXP_Out.exp") — vipw, Feb 22 '13 at 14:31
This python can be improved a lot. I can give you better output if you give me the input format of your files. — ATOzTOA, Feb 22 '13 at 14:34
Does this help? http://stackoverflow.com/questions/89228/calling-an-external-command-in-python — rainer, Feb 22 '13 at 14:34
Posted this below, but issue was not having full path. Works now. Reason for calling from python is because it is part of a larger script that needs to run. — Chris Hanes, Feb 22 '13 at 16:20
I can't post the code (site wants me to wait 10 hours), but it ran in 5 minutes (rather than 6 hours) — Chris Hanes, Feb 22 '13 at 19:28

ATOzTOA · Answer 1 · 2013-02-22T16:48:31.787

Here is a optimized version of pure python code:

def main(Output_CSV, Exp_Merge, updated_exp):
    output_list = []

    # copy header records
    records = open(Exp_Merge,'r').readlines()
    output_list = records[0:2]

    serials = open(Output_CSV,'r').readlines()
    serials = [x.split(",")[2].strip().rstrip('\n') for x in serials]

    for s in serials:
        items = [x for x in records if s in x]
        output_list.extend(items)

    open(updated_exp, "w").write("".join(output_list))

main("sn.txt", "merge_exp.exp", "outx.txt")

Input

sn.txt:

x,y,0011
x,y,0002

merge_exp.exp:

Header1
Header2
0011abc
0011bcd
5000n
5600m
6530j
0034k
2000lg
0002gg

Output

Header1
Header2
0011abc
0011bcd
0002gg

Try this out and see how much time it takes...

I actually got it to work. The issue is that without the full path to grep, it fails. If you run in command line outside the grep.exe location, it gives you the "regular expression too long" error. Python does not give you that. Once I put in the full path to grep it worked great: — Chris Hanes, Feb 22 '13 at 16:18

score 0 · Accepted Answer · answered Feb 25 '13 at 17:52

When I use full path to grep location it worked (I pass it the grep_loc, Serial_List, Export):

import os

Export_Dir = os.path.dirname(Export)
Export_Name = os.path.basename(Export)

Output = Export_Dir + "\Output_" + Export_Name
print "\nOutput: " + Output + "\n"

cmd = grep_loc + " --file=" + Serial_List + " " + Export + " > " + Output
print "grep usage: \n" + cmd + "\n"
os.system(cmd)
print "Output created\n"

score 0 · Answer 3 · answered Oct 08 '13 at 07:07

I think you have not chosen the right title for your question: What you want to do is the equivalent of a database JOIN. You can use grep for that in this particular instance, because one of your files only has keys and no other information. However, I think it is likely (but of course I don't know your case) that in the future your sn.txt may also contain extra information.

So I would solve the generic case. There are multiple solutions:

import all data into a database, then do a LEFT JOIN (in sql) or equivalent
use a python large data tool

For the latter, you could try numpy or, recommended because you are working with strings, pandas. Pandas has an optimized merge routine, which is very fast in my experience (uses cython under the hood).

Here is pandas PSEUDO code to solve your problem. It is close to real code but I need to know the names of the columns that you want to match on. I assumed here the one column in sn.txt is called key, and the matching column in merge_txt is called sn. I also see you have two header lines in merge_exp, read the docs for that.

# PSEUDO CODE (but close)
import pandas
left = pandas.read_csv('sn.txt')
right = pandas.read_csv('merge_exp.exp')
out = pandas.merge(left, right, left_on="key", right_on="sn", how='left')
out.to_csv("outx.txt")

Use grep on file in Python

3 Answers3