Faster than Grep on Python?

Question

So, the bash command that I'd be normally using in a Bash Script would be something like:

$ cat huge2GBfile.txt | grep -w "pattern1/|pattern2/|pattern3" > out.txt

It will output the lines in huge2GBfile where it has found pattern1,2,3. I was wondering if this is achievable through python. I know that I can use

os.system(cmd)

But I'd like to know if there is something similar in Python (I am a complete noob) and if it is faster than using cat+grep. Thanks!

Initial thoughts, would something like

for line in f:
     if pattern in line:
          out.write(line)

be faster?

sidenote: `cat X | grep Y` can be written shorter as `grep Y X` ([UUOC](http://partmaps.org/era/unix/award.html)) — miku, Oct 05 '13 at 21:30
And `cat X | anytool` can be written generically as `anytool < X`. Why people use cat when not catenating anything is a mystery. — Wooble, Oct 05 '13 at 21:32
I doubt you can get it to go faster than grep, but there is a `re` module for regular expressions in Python. — Cameron, Oct 05 '13 at 21:32
grep's regex engine has been highly optimized over many years and uses a better design than the python re module to begin with. — Wooble, Oct 05 '13 at 21:35
What I ended up doing is by adding LC_ALL=C in front of the command. It turns the search type from UTF8 to ASCII which gave me 140x better performance. — Mojing Liu, Nov 05 '13 at 01:36

score 4 · Answer 1 · edited May 23 '17 at 10:31

Even with an algorithm that is better than the logic grep uses (as someone already commented they are highly optimised, grep is 30 years old!), there is still the fact that they are utilities written in C, and compiled natively for the system.

Python is an interpreted language, and can be a couple of orders of magnitude slower than native C, so I would argue that the answer is no, there is nothing in python that could be faster.

If you want to process the output of a grep command line by line, an option would be to build your python script similar to a unix command line tool, so that it can read from stdin and write to stdout, so you could use something like :

grep pattern file | python myscript.py

How do you read from stdin in Python?

Faster than Grep on Python?

1 Answers1