2

I am trying to use fabric 2.3 to run few tasks that require sudo on some servers . My main goal here was to parallelize the operation so i thought of using ThreadingGroup class of fabric api however it does not support sudo.

Below is my code for sake of clarity

#!/usr/bin/env python

from fabric import ThreadingGroup, Config
from getpass import getpass


sudo_pass = getpass("Enter your sudo password: ")
sudo_config = Config(overrides={'sudo': {'password': sudo_pass}})
server_pool = ThreadingGroup("test1", "test2",
                             config=sudo_config)

result = server_pool.sudo('cat /etc/shadow', hide='stderr')
print(result)

Now this does not work as mentioned above because ThreadingGroup does not support all of the methods that Connection class support.

I can run sudo on multiple servers by iterating over the individual hosts and then creating connection for each but that isn't efficient.

So is there a way to make this parallel with fabric 2.3 ? I have gone through the official documentation as well but did not find anything.

Further i did some more testing on it following the official documentation and it seems like ThreadingGroup achieves parallelism only if you run it like below

fabric.ThreadingGroup('test1', 'test2').run('uname -s')

however if you run it like below it does not run in parallel

def run_task(c):
    c.run('uname -s')


for cxn in fabric.ThreadingGroup('test1', 'test2'):
    run_task(cxn)

So it looks like there isn't much flexible support for parallelism in fabric 2.3 as of now and i might have to switch back to fabric version 1 only.

Rohit
  • 3,659
  • 3
  • 35
  • 57
  • If you want to run everything as root why not execute the script with sudo from the start? – Hultner Sep 06 '18 at 08:32
  • 1) I am not trying to run everything as root, as there are only certain set of tasks that require sudo. 2) How would running the script from my local machine as `sudo` would grant the `sudo` on remote machines ? 3) What if i have another group of remote hosts that doesn't require `sudo` at all ? – Rohit Sep 06 '18 at 08:38
  • You can set the [`NOPASSWD`](https://askubuntu.com/questions/334318/sudoers-file-enable-nopasswd-for-user-all-commands) attribute in `/etc/sudoers` for your user and that command. That way you're explicitly whitelisting the command you want to run in `fabric`, and your password is not in danger of leaking. – Nils Werner Sep 06 '18 at 09:27
  • @NilsWerner This is not really related to fabric at all, rather more related to configuring sudo on the system itself. My question is more on how to run sudo in parallel with new version of fabric !! i am not at any worry of leaking the password, since i am inputting it with getpass and password or no password is entirely separate issue. – Rohit Sep 06 '18 at 09:30
  • Once that password prompt is gone, replace `server_pool.sudo('cat /etc/shadow', hide='stderr')` with `server_pool.run('sudo cat /etc/shadow', hide='stderr')`. – Nils Werner Sep 06 '18 at 09:30
  • This is a workaround not a solution. Also one may not have the leverage to get run `sudo` with `NOPASSWD` on all systems they want to run something. So for me this is not even workaround. – Rohit Sep 06 '18 at 12:56

1 Answers1

1

It seems some functions are not implemented in current version (2.4 so far). An optional solution is to add some code in its source file. You can find the install path of fabric, and edit group.py.

First, add this function in group.py:

def thread_worker_sudo(cxn, queue, args, kwargs):
    result = cxn.sudo(*args, **kwargs)
    # TODO: namedtuple or attrs object?
    queue.put((cxn, result))

and then add sudo function in class ThreadingGroup:

class ThreadingGroup(Group):
    .... original ThreadingGroup


    def sudo(self, *args, **kwargs):
        results = GroupResult()
        queue = Queue()
        threads = []
        for cxn in self:
            my_kwargs = dict(cxn=cxn, queue=queue, args=args, kwargs=kwargs)
            thread = ExceptionHandlingThread(
                target=thread_worker_sudo, kwargs=my_kwargs
            )
            threads.append(thread)
        for thread in threads:
            thread.start()
        for thread in threads:
            # TODO: configurable join timeout
            # TODO: (in sudo's version) configurability around interactive
            # prompting resulting in an exception instead, as in v1
            thread.join()
        # Get non-exception results from queue
        while not queue.empty():
            # TODO: io-sleep? shouldn't matter if all threads are now joined
            cxn, result = queue.get(block=False)
            # TODO: outstanding musings about how exactly aggregate results
            # ought to ideally operate...heterogenous obj like this, multiple
            # objs, ??
            results[cxn] = result
        # Get exceptions from the threads themselves.
        # TODO: in a non-thread setup, this would differ, e.g.:
        # - a queue if using multiprocessing
        # - some other state-passing mechanism if using e.g. coroutines
        # - ???
        excepted = False
        for thread in threads:
            wrapper = thread.exception()
            if wrapper is not None:
                # Outer kwargs is Thread instantiation kwargs, inner is kwargs
                # passed to thread target/body.
                cxn = wrapper.kwargs["kwargs"]["cxn"]
                results[cxn] = wrapper.value
                excepted = True
        if excepted:
            raise GroupException(results)
        return results

I just copy the code of function run and replace this line 

thread = ExceptionHandlingThread(
    target=thread_worker, kwargs=my_kwargs
)

with 

thread = ExceptionHandlingThread(
    target=thread_worker_sudo, kwargs=my_kwargs
)

It works for me like that:

def test_sudo(group):
    group.sudo('whoami', user='root')

$ python fabfile.py
root
root
root
root
root

However, I am not sure, it will work well for all situations.

iuyoy
  • 121
  • 1
  • 6