One asyncio program to manage many others

Question

Recently I've been experimenting with discord bots, which use asyncio. I've been in the process of making a program that controls lots of other bots, opening and closing them on the fly, but I have a problem; I've experimented with subprocess.Popen, rpyc and multiprocessing, but I'm struggling to work out how to have communication between programs. I've tried using the following line of code to launch the subprocesses:

Popen('python smallprogram.py', stdout=PIPE, stdin=PIPE)

but I'm still unable to communicate from the main program to the smaller programs, due to the smaller ones needing to run asyncio. This prevents me from using input() with Popen.communicate(). Ideally, I'd like a way to call a function on the smaller program when needed, with the small program still running asyncio. I don't mind having to paste the same block of code into each of the smaller programs, but I think this might also be solvable with some importations?

Can this be done? I've never made an API before, but it seems I might need to use API as a template. Thank you :)

Note: I only need to do big->small communication, but doing it the other way round would be nice too.

Why not simply spawn new asyncio tasks instead of using subprocesses? — Vincent, Aug 20 '17 at 19:15
@Vincent I'm not too familiar with how I'd go about doing this. The idea of the program is that I can modify the bots on the fly, meaning that I can modify one's code without restarting the others, as well as manually injecting code. Wouldn't this require having the bots pre-written beforehand? — Bruno E, Aug 21 '17 at 09:34

Vincent · Accepted Answer · 2017-08-21T14:37:37.923

There many ways to deal with inter-process communication, and I think your use of stdin/stdout is a valid approach.

It turns out it's possible to read asynchronously from stdin in asyncio, though it's quite tricky to do using only the standard library.

Alternatively, you can use the aioconsole helper ainput:

import aioconsole

async def echo_child():
    data = await aioconsole.ainput()
    print(data, end='')

Or get_standard_streams for an interface similar to the stream API:

import aioconsole

async def echo_child():
    stdin, stdout = await aioconsole.get_standard_streams()
    data = await stdin.readline()
    stdout.write(data)

On the parent side:

import asyncio

async def parent():
    proc = await asyncio.create_subprocess_exec(
        sys.executable, 'child.py',
        stdin=asyncio.subprocess.PIPE,
        stdout=asyncio.subprocess.PIPE)
    proc.stdin.write(b'Hello\n')
    data = await proc.stdout.readline()
    print(data.decode().strip())
    proc.terminate()

One asyncio program to manage many others

1 Answers1