grep No such file or directory with envoy.run

Question

I try to build script that execute some grep search in my logs and print the results. I try to use Envoy because is more easy than subprocess but when I execute grep command it gives me back an error of no such file o directory.

The dir structure is easy:

• . # root of script
• test.py # script file
• web_logs/log/ # dir that contains log to search in

My test.py is easy:

import envoy

def test(value):

   search = "grep 'cv="+str(value)+"' ./web_logs/log/log_*"
   print(search) #check of the search string
   r = envoy.run(search)
   print(r.status_code, r.std_out, r.std_err)#check of the command
   response = r.std_out

if __name__ == "__main__":
   test(2)

The output is:

grep 'cv=2' ./web_logs/log/log_*
(2, '', 'grep: ./web_logs/log/log_*: No such file or directory\n')

If i run the same command:

grep 'cv=2' ./web_logs/log/log_*

I can find the occurrence of the string "cv=2" in the log files.

Where is the error?

Update after the answers The problem is in using of * that envoy cannot explode without use of glob module so I using the subprocess as it is and I try to study better the using of glob module to improve envoy.

The new code I used is:

import subprocess

def test(value):

   search = "grep 'cv="+str(value)+"' ./web_logs/log/log_*"
   print(search) #check of the search string
   proc = subprocess.check_output(search, shell=True)
   print proc.split('\n')

if __name__ == "__main__":
   test(2)

Answer 1

@baptistemm is actually right in that since you're not running bash as part of your process the globbing is not working.

However what's happening is a bit deeper.

When you run a sub process it can be done by one of several system services (system calls).

Short Answer (TLDR;)

Here's the correct way to do this:

import envoy

def test(value):

   search = "/bin/sh -c \"grep 'cv="+str(value)+"' ./web_logs/log/log_*\""
   print(search) #check of the search string
   r = envoy.run(search)
   print(r.status_code, r.std_out, r.std_err)#check of the command
   response = r.std_out

if __name__ == "__main__":
   test(2)

Running the command as a shell command will take care of globbing.

Long answer

Whenever a sub process is executed, it eventually gets translated into an execve system call (or equivalent).

In C library there're helper functions such as system(3) and popen(3) which wrap around execve(2) to provide easier ways of executing processes. system launches a shell and passes its argument as is to -c option of the shell. popen does extra magic, kinda like what envoy is doing in python.

In envoy, the argument is parsed for | (see def expand_args(command):) in the envoy code. and then uses the equivalent of popen to execute the processes. envoy is essentially what the shell does with the | marker (splits things up across the | and then uses popen).

What envoy is NOT doing is interpreting * as the shell does, as in expanding it to match files using a glob function of some sort. Bash does. Thus my answer.

A fun exercise would be for you to contribute code to envoy :-) and make it do the globbing.

Answer 2

Why it works in terminal but not in envoy is related to globbing (bash example).

When you run in your terminal

grep 'cv=2' ./web_logs/log/log_*

bash will parse the command line and replace the star character with every occurence of file that match. So if you have ./web_logs/log/log_1 ./web_logs/log/log_2 and ./web_logs/log/log_foo your command will actually be

grep 'cv=2' ./web_logs/log/log_1 ./web_logs/log/log_2 ./web_logs/log/log_foo

When you execute the same thing in envoy, that'd different, it won't perform the globing of the files then it'll pass to grep a file named ./web_logs/log/log_* which doesn't exist, this is actually confirmed by the line you pasted in your question.

print r.std_err
'grep: ./web_logs/log/log_*: No such file or directory\n'

ps: there is a glob module for python