Run python script on all subdirectories with specific name

Question

I have a python script which takes some parameters, and I want to run this script on all the sub directories starting from the one that the script is inside.

The idea is I want to get a custom output of the script to be saved in the file.

here's what i've done:

for x in `find . -type d | grep data`;
do
python /full/path/to/file/script.py -f "%a
%t" $x/*.txt -o $x/res.txt
done

But this is not working and I don't know why. The grep in the for loop is to only get the directories that contains .txt files and apply the script on them.

The new line between %a and %t is because I want to customize the output of the output of the python script to include a new line between each 2 variables

What am I doing wrong ?

Could you be more specific? You want to run the python script in all subdirectories of the directory it is in? Or you want to run it generically on whatever files are passed to it? It it's the first then `os.walk` would be a better way to do it that passing it all these files. — will, Feb 21 '13 at 15:22
You say that your script is not working. What is the output and how is it different from the expected output? — imp25, Feb 21 '13 at 15:33
@will i want to run the script in all subdirectories of the directory it is in .. which only match the directory name"data" — ifreak, Feb 21 '13 at 15:34
@imp25 it's returning the error that it cannot find the file or directory that i'm passing as an argument for the python script, even thought they are found, but looks like it has a problem with path starting with `./path/to/directory` — ifreak, Feb 21 '13 at 15:36

will · Answer 1 · 2013-02-21T16:15:22.490

0

If you want to run this script on all the sub directories starting from the one that the script is inside, then try doing it this way instead:

import os

for path, directories, files in os.walk(os.path.dirname(os.path.realpath(__file__))):
    print path, directories, files
    txt_files = [arbitrary_file for arbitrary_file in files if arbitrary_file[-4:].lower() == ".txt"]

    #run your python here
    txt_files = [txt_file for arbitrary_file in files if arbitrary_file[]

If your original code was this:

import sys

text_files_to_process = #Do Something with sys.argv - or whatever you're using to parse your arguments.

with open("res.txt", "w") as f:
    #do something with all the text files, and write the output to res.txt.
    for text_file in text_files_to_process:
        with open(text_file) as tf:
            for line in tf:
                #whatever your text processing is
            tf.write("something")

then you just alter it to something like this:

import os

for path, directories, files in os.walk(os.path.dirname(os.path.realpath(__file__))):
    print path, directories, files
    txt_files = [arbitrary_file for arbitrary_file in files if arbitrary_file[-4:].lower() == ".txt"]

    txt_files = [txt_file for arbitrary_file in files if arbitrary_file[]

    with open("res.txt", "w") as f:
        #do something with all the text files, and write the output to res.txt.
        for text_file in txt_files:
            with open(text_file) as tf:
                for line in tf:
                    #whatever your text processing is
                tf.write("something")

edited Feb 21 '13 at 16:15

answered Feb 21 '13 at 15:33

will

10,260
6
46
69

the script will calculate the result for all the .txt files inside each sub-directory (data) not for each txt file – ifreak Feb 21 '13 at 15:41
Yes i realised this. Your program accepts a list of text files for each sub directory. That is what the `txt_files` list is. It will be overwritten for each new subdirectory `os.walk` enters. – will Feb 21 '13 at 15:44
where does my python call become in this case.. and how can i give it a parameter as a list ?? – ifreak Feb 21 '13 at 15:50
Well you could use one of the functions to run your other script (something like `eval`, or `exec`). But it would be better to add this in at the beginning of your script and put the original inside the loop. – will Feb 21 '13 at 15:53
the problem that the script i'm running is not mine .. i'm just using it as free source .. i will try your suggestion .. – ifreak Feb 21 '13 at 15:55
i did not know where to put the `txt_files` list inside the `eval` .. that's confusing .. can you please write an understandable example that will run the code ? – ifreak Feb 21 '13 at 16:04
let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/24912/discussion-between-ifreak-and-will) – ifreak Feb 21 '13 at 16:14
ok, now i managed to run the script inside the other one as you suggested, but the problem is that i want to customize the output of the other script printing a new line between `%a` and `%t` and this making a problem, how can i avoid this ? – ifreak Feb 21 '13 at 16:45

Run python script on all subdirectories with specific name

1 Answers1

Linked