1

I used xargs for parallel processing. This thread follows what I had. Parallel processing with xargs in bash But the parallel processing increased CPU load. I was running the script for 57 openstack tenants to fetch results from them. This report is running every 2 hours which is causing CPU spike.

To decrease the load, I thought of adding the random sleep time...something like below but it didn't do much help. I am not sure if I can use NICE to set a priority because the results I am fetching are from openstack server. If it's possible, please let me know.

I could remove the parallel processing but this will take 6 hours for me to get all the reports from tenants. So that is not an option.

If there is any other way to optimize this...any ideas or suggestions would be great. 

source ../creds/base
printf '%s\n' A B C D E F G H I J K L M N O P |

xargs -n 3 -P 8 bash -c 'for tenant; do
      source ../creds/"$tenant"
      python ../tools/openstack_resource_list.py "$tenant"> ./reports/openstack_reports/"$tenant".html
      sleep $[ ( $RANDOM % 10 )  + 1 ]s
done' _

Python file

with open('../floating_list/testReports/'+tenant_file+'.csv', 'wb') as myfile:
    fields = ['Name', 'Volume', 'Flavor', 'Image', 'Floating IP']
    writer = csv.writer(myfile)
    writer.writerow(fields)
    try:
        for i in servers:
            import io, json
            with open('../floating_list/testReports/'+tenant_file+'.json', 'w') as e:
                e.write(check_output(['openstack', 'server', 'show', i.name, '-f', 'json']))
            with open('../floating_list/testReports/'+tenant_file+'.json', 'r') as a:
                data = json.load(a)
                name.append(i.name)
                volume = data.get('os-extended-volumes:volumes_attached', None) 
                if volume:
                    vol = [d.get('id', {}) for d in volume if volume]
                    vol_name_perm = []
                    for i in vol:
                        try:
                            vol_name1 = (check_output(['openstack', 'volume', 'show', i,  '-c', 'name', '-f', 'value'])).rstrip()
                            vol_name_perm.append(vol_name1)
                        except:
                            vol_name_perm.append('Error')
                    vol_join = ','.join(vol_name_perm)
                    vol_name.append(vol_join)
                else:
                    vol_name.append('None')
                ...

        zipped = [(name), (vol_name),(flavor),(image),(addr)]
        result = zip(*zipped)
        for i in result:
            wr = csv.writer(myfile, quoting=csv.QUOTE_ALL)
            wr.writerow(i)
    except (CalledProcessError, IndexError) as e:
        print (e)
        print("except", i.name)

WITH GNU PARALLEL

printf '%s\n' A B C D E F | parallel --eta -j 2 --load 40% --noswap 'for tenant; do
      source ../creds/"$tenant"
      python ../tools/openstack_resource_list.py "$tenant"> ./reports/openstack_reports/"$tenant".html
done'

I get syntax error near unexpected token `A'

WORKAROUND

I am able to manage the load with xargs -n 1 -p 3 for now. This gives me reports within 2 hours. I still want to explore my options with GNU Parallel as suggested by Ole Tange

Heenashree Khandelwal
  • 659
  • 1
  • 13
  • 30
  • 1
    Like I commented on your other question just now, optimizing the internals of `openstack_resouce_list.py` would seem like the most fruitful option to explore, but we can't help you with that without seeing the code and understanding what it's supposed to do. – tripleee Jan 30 '18 at 05:36
  • If you can have each tenant report its status, say, every 5 minutes into a central server, generating the report would get you results which are fresh to the latest 5 minutes without any significant load from compiling the report itself. – tripleee Jan 30 '18 at 05:38
  • Have you tried to replace `-n 3` with `-n 1`? – Cyrus Jan 30 '18 at 05:39
  • 2
    **GNU Parallel** would be pretty much a *"drop in replacement"* for `xargs` and it has the new `--load` and `--limit` options which throttle jobs according to CPU load.... – Mark Setchell Jan 30 '18 at 08:13
  • @Cyrus, -n 1 did not work. – Heenashree Khandelwal Jan 30 '18 at 12:02
  • @MarkSetchell, I definitely will try this one GNU Parallel – Heenashree Khandelwal Jan 30 '18 at 12:02
  • @MarkSetchell, I worked on it, does my edit make sense? I read about GNU and still understanding. – Heenashree Khandelwal Jan 31 '18 at 09:34
  • https://docs.openstack.org/python-novaclient/pike/ would allow you to at least avoid the overhead of running `openstack` commands in a subprocess for each query. If you want to report more or less the full state of each instance, running just two or three queries (servers, volumes, groups?) and parsing the results you get back into your own data structures will likely offer some improvements if the overhead for each call is still significant. – tripleee Jan 31 '18 at 09:42
  • Forgetting all the parallel stuff for a moment, can you show the first 2 commands that need running just on their own please? I am failing to see the point of your `for` loop... – Mark Setchell Jan 31 '18 at 10:06
  • I source into each file which has openstack creds, and A B C that you see are the tenants themselves. (57 tenants that I have). so for loop goes over each and every tenant and generate a report. – Heenashree Khandelwal Jan 31 '18 at 10:15
  • 1
    @MarkSetchell The linked earlier question from the OP probably provides the background you are looking for. – tripleee Jan 31 '18 at 10:20
  • but I doubt if we cannot control the load of CPU and memory of a remote server from GNU Parallel. – Heenashree Khandelwal Jan 31 '18 at 12:01
  • @tripleee, I must use openstack-clients now, that was one resort I had hoped could help me. – Heenashree Khandelwal Jan 31 '18 at 12:02
  • Can you change the question so it is much more explicit that the server which becomes loaded is a _remote_ server and _not_ the local machine? – Ole Tange Feb 02 '18 at 07:21

1 Answers1

0

Maybe you can use GNU Parallel with niceload (part of GNU Parallel):

niceload parallel --nice 11 --bar "'source ../creds/{};
  python ../tools/openstack_resource_list.py {} > ./reports/openstack_reports/{}.html'" ::: {A..F}
Ole Tange
  • 31,768
  • 5
  • 86
  • 104