1

I have a python loop that at each iteration is creating a new image in a new path. I want to send that path into a process previously executed that is waiting for a path at the command prompt.

In more detail:

I am running a self-driving simulator in python and every 5 frames I want to test that current frame (that is saved in a RAM disk) into an object detector algorithm I have trained (that spends arround 9ms to detect my object BUT 2 seconds to open the process). Actually, I execute the trained algorithm by using the subprocess module, but the problem I have is that when that process is opened (I just open the process once, when I run the main script) it is waiting for a path image. I believe that with your tips I am close to the answer but I don't face how to pass that path image to this subprocess that is waiting for it at each 5 frames iteration.

PD: I am on Windows, Python 3.5.4

Do you know what can I do?

Martí Sánchez Juanola
  • 27
  • 3
  • 2
    Can you describe this in more detail? Maybe add code and tell us what you tried and what didn't work? – Christian Macht Jun 28 '19 at 17:21
  • 1
    Check out the [`subprocess`](https://docs.python.org/3/library/subprocess.html) library. It's fairly useful for tasks like this – C.Nivs Jun 28 '19 at 17:24
  • 1
    Yes, I have a python loop that at each iteration it is creating a new image in a new path. I want to send that path into a process previously executed that is waiting for a path at the command prompt. – Martí Sánchez Juanola Jun 28 '19 at 17:26
  • 1
    Because you didn't give us some code I can think of several different solutions: [1. shared variable multiprocessing](https://stackoverflow.com/questions/17377426/shared-variable-in-pythons-multiprocessing) , [2. Async IO](https://realpython.com/async-io-python/) , [3. global variable between threads](https://stackoverflow.com/questions/19790570/using-a-global-variable-with-a-thread) – Omri Attiya Jun 28 '19 at 17:39
  • The easiest way is probably to restructure your code so you don't have to do it that way. If you have can edit the code of the process that's waiting for input, you can redirect stdin, as in [one of these answers](https://stackoverflow.com/questions/31239002/enter-to-raw-input-automatically) –  Jun 28 '19 at 17:53
  • What platform are you on? – CryptoFool Jun 28 '19 at 18:20
  • Have a look at subprocess.Popen.stdout. There are various answers to such usecases around. – MisterMiyagi Jun 28 '19 at 18:36
  • I am running a self-driving simulator in python and every 5 frames I want to test the current frame (that is saved in a RAM disk) into an object detector algorithm I have trained (that spends arround 9ms to detect my object BUT 2 seconds to open the process). Actually, I execute the trained algorithm by using the subprocess module, but the problem I have is that when that process is opened (I just open the process once) it is waiting for a path image. I believe that with your tips I am close to the answer but I don't face how to pass a path image to this subprocess that is waiting for it. – Martí Sánchez Juanola Jun 29 '19 at 08:07
  • Seems to me that either you can link these two processes together by having the one process reading the output from the other, or you can't. If you can, then you can follow up with information provided so far here. If you need an independent process running ahead of time that is totally independent of the process producing the image names, then you can't do this. At that point, you're talking about some sort of client/server model, of which there are many. Even if you ran both the client and server in the same process space, if the two are independent, it's still client/server... – CryptoFool Jun 29 '19 at 17:59
  • For client/server, the image name generator is the client, and the waiting subprocess is the server. You need to pick a way to allow the server to present an API that allows the client to pass it image names. There are many options here, just which ones being largely dependent on where the two processes with respect to each other. If they're in the same process space, you could just use a shared queue/list structure. If they're in separate processes on the same machine, you could use some sort of file IO. Or... – CryptoFool Jun 29 '19 at 18:03
  • you could have the server represent a HTTP REST interface that allowed any HTTP Client to make a request handing it an image name/path. With this model, your two processes could be running on different hosts. If that were the case, you'd have a complication with passing a filename and having it be valid on the server. But it sounds like you want this all running on the same machine at least. So then just pick some mechanism to allow the server to run and watch for incoming image paths: shared data structure, shared file, HTTP API, shared memory system (redis), message queue (RabbitMQ)... – CryptoFool Jun 29 '19 at 18:07
  • Thank you for all your comments and for your help. Finally I have achieved it. I use the library 'pexpect' that allows to launch a program inside a python script with the function process = popen_spawn.PopenSpawn. Then, a function from that library called 'send' allows to pass an argument to that running process (process.send(arg)). In my case, I launch the program (.exe) at the beginning of the script defining that instance as a global variable. Then I just have to execute the send function at each iteration. – Martí Sánchez Juanola Jul 01 '19 at 11:50

2 Answers2

1

If you're on a *nix environment, and I'm understanding what you want, pipes provides what you want to do quite elegantly. I don't know Windows, but maybe the same concept could be used there.

Here's a simple example that illustrates this:

Pipe1a.py:

#!/usr/bin/env python

import time
import sys

for i in range(10):
    time.sleep(1) # represent some processing delay
    print("filename{}.jpg".format(i))
    sys.stdout.flush()

Pipe1b.py:

#!/usr/bin/env python

import sys

while True:
    line = sys.stdin.readline()
    if len(line) == 0:
        break
    print "Processing image '{}'".format(line.strip())

If you made both of these scripts executable, then you could chain them together via a pipe at the command prompt:

> Pipe1a.py | Pipe1b.py

Resulting output:

Processing image 'filename0.jpg'
Processing image 'filename1.jpg'
Processing image 'filename2.jpg'
Processing image 'filename3.jpg'
Processing image 'filename4.jpg'
Processing image 'filename5.jpg'
Processing image 'filename6.jpg'
Processing image 'filename7.jpg'
Processing image 'filename8.jpg'
Processing image 'filename9.jpg'

The concept here is that one process writes data to its stdout and a second process reads that data from its stdin.

If you don't want to use the command prompt to string these together, but rather want to run a single script, you can do this same thing, with pretty much the same code, using the Python subprocess1 module. For this example, you could have the Pipe1b.py program run the Pipe1a.py program via subprocess.Popen and then process its output in this same way.

CryptoFool
  • 21,719
  • 5
  • 26
  • 44
0

Thank you for all your comments and for your help. Finally I have achieved it.

I use the library 'pexpect' that allows to launch a program inside a python script with the function process = popen_spawn.PopenSpawn. Then, a function from that library called 'send' allows to pass an argument to that running process (process.send(arg)).

In my case, I launch the program (.exe) at the beginning of the script defining that instance as a global variable. Then I just have to execute the send function at each iteration.

Martí Sánchez Juanola
  • 27
  • 3