os.walk find a directory which is not even present

Question

I need to extract some values out of a file, i wrote the following code.

import os
import sys
rootdir='/home/nsingh/ansible-environments/aws'
for root, subdirs, files in os.walk(rootdir):
  for j in subdirs:
    print j
    mypath=rootdir+'/'+j+'/inventory/group_vars/all'
    #print mypath
    fo=open(mypath,'r')
    f=fo.readlines()
    for line in f:
            if ('isv_alias' in line or 'LMID' in line or 'products' in line):
                         path='/home/nsingh/krikCSV_fun.csv'
                         if('isv_alias' in line):
                            line=line.strip('isv_alias=')
                            line= line.strip('"')
                         elif('LMID'  in line):
                            line=line.strip('LMID=')
                         else:
                            line=line.strip('products=')

                         fi= open(path,'a+')
                         fi.write(line)
                         fi.close()
    fo.close()

the os.walk method somehow finds a hidden directory as well which is not actually present

loadgen
crapcity
rmstest2
suricatatest
.git
Traceback (most recent call last):
  File "testme.py", line 9, in <module>
    fo=open(mypath,'r')
IOError: [Errno 2] No such file or directory: '/home/nsingh/ansible-environments/aws/.git/inventory/group_vars/all'

OUTPUT:

: "suricatatest"^M
: suricatatest
: rms_ems_hosted
: 26
: rmstest2
: rms_scl
: 80
: suricatatest
: rms_ems_hosted
: 26
: "suricatatest"^M
: suricatatest
: rms_ems_hosted
: 26

I need the output as & also remove the semicolon:

suricatatest rms_ems_hosted 26

Answer 1

What makes you think that /.git doesn't exist?

Try this:

import os

rootdir = '/home/nsingh/ansible-environments/aws'
for root, subdirs, files in os.walk(rootdir):
    for j in subdirs:
        print(j)
        my_path = rootdir + '/' + j + '/inventory/group_vars/all'
        if os.path.isfile(my_path):
            with open(my_path, 'r') as fo:
                for line in fo.readlines():
                    if 'isv_alias' in line or 'LMID' in line or 'products' in line:
                        path = '/home/nsingh/krikCSV_fun.csv'
                        if 'isv_alias' in line:
                            line = line.strip('isv_alias=')
                            line = line.strip('"')
                        elif 'LMID' in line:
                            line = line.strip('LMID=')
                        else:
                            line = line.strip('products=')

                        with open(path, 'a+') as fi:
                            fi.write(line.lstrip(": "))

Answer 2

You should use os.path to make the file paths. os.walk will visit all the directories in the tree under the top directory - you are only interested in directories that end with 'inventory/group_vars', so check for that and take action. If you want to write the values as a group, you need to collect them in something.

import os, os.path, collections
rootdir = '/home/nsingh/ansible-environments/aws'
sub_folder = 'inventory/group_vars'
out_path = '/home/nsingh/krikCSV_fun.csv'
for dirpath, dirnames, filenames in os.walk(rootdir):
    if dirpath.endswith(sub_folder):
        data = collections.defaultdict(list)
        with open(os.join(dirpath, 'all')) as f, open(out_path, 'a+') as out:
            for line in f:
                if 'isv_alias' in line:
                    line = line.strip('isv_alias=')
                    line = line.strip('"')
                    data['isv_alias'].append(line)
                elif 'LMID'  in line:
                    line = line.strip('LMID=')
                    data['LMID'].append(line)
                elif 'products' in line:
                    line = line.strip('products=')
                    data['products'].append(line)
            for a, b, c in zip(*data.values()):
                out.write('{},{},{}\n'format(a, b, c))

I used a defaultdict to store multiple items of interest from a single file. If there is only one 'isv_alias', 'LMID', 'products' group in each file then you could just as easily store the information in a list or namedtuple.

---

You didn't provide an example of the file(s) so it's not clear what the line structure is. If it looks like this:

isv_alias="foo"
LMID=bar
products=26

It can be simplified to

keys = {'isv_alias', 'LMID', 'products'}
for dirpath, dirnames, filenames in os.walk(rootdir):
    if dirpath.endswith(sub_folder):
        data = collections.defaultdict(list)
        with open(os.join(dirpath, 'all')) as f, open(out_path, 'a+') as out:
            for line in f:
                line = line.strip()
                key, value = line.split('=')
                if key in keys:
                    value = value.strip('"')
                    data[key].append(value)
            for a, b, c in zip(*data.values()):
                out.write('{},{},{}\n'format(a, b, c))

---

As long as you are accumulating the information in data, you can just open the output file once

data = collections.defaultdict(list)
keys = {'isv_alias', 'LMID', 'products'}
for dirpath, dirnames, filenames in os.walk(rootdir):
    if dirpath.endswith(sub_folder):
        with open(os.join(dirpath, 'all')) as f:
            for line in f:
                line = line.strip()
                key, value = line.split('=')
                if key in keys:
                    value = value.strip('"')
                    data[key].append(value)

with open(out_path, 'a+') as out:
    for a, b, c in zip(*data.values()):
        out.write('{},{},{}\n'format(a, b, c))

---

If using Python 3.6 or an ordered defaultdict then the solution above assumes the order of appearance of each key in the file is the order you want them written out.

If the file structure isn't ordered or the dictionary used isn't ordered, write to the file like this:

            for a, b, c in zip(data['isv_alias'], data['LMID'], data['products']):
                out.write('{},{},{}\n'format(a, b, c))