2

I would like to create a simple Python program that will concurrently execute 2 independent scripts. For now, the two scripts just print a sequence of numbers but my intention is to use this program to concurrently run a few Twitter streaming programs in the future.

I suspect I need to use subprocess.Popen but I cannot quite get my head around what arguments I should put in there. There was a similar question on StackOverflow but the code provided there (pasted below) doesn't print anything. I will appreciate your help.

My files are:

thread1.py
thread2.py

import subprocess

subprocess.Popen(['screen', './thread1.py']))
subprocess.Popen(['screen', './thread2.py'])
user3722736
  • 191
  • 1
  • 2
  • 12
  • 1
    Isn't it simpler and more obvious to run parallel processes using pipelines in Unix? I.e. ``./thread1.py;./thread2.py`` – wanderlust Jun 11 '14 at 13:20
  • Yes, normal Unix process pipeline will do the trick. Whereas just a heads up, if the operations are i/o bound you should consider using threads over processes in Python. – Aashish P Jun 11 '14 at 13:22
  • Thanks, I've tried what you suggested: python ./thread1.py;./thread2.py but I'm only getting the printed numbers from the first script and I would like the two scripts to be running at the same time. – user3722736 Jun 11 '14 at 13:46
  • 1
    Because of the pipeline usage you should run both your scripts as separate python programs: ``python thread1.py;python thread2.py``. In your case I assume you don't have the python preambula, so your scripts require to use python interpreter, as I stated above. That's the reason, why you second script didn't run. Pay attention you should be in the directory your files are located in, and you don't need the "./" prefix, that indicates you to take file from the present folder. – wanderlust Jun 11 '14 at 14:10
  • I've tried to use the above Unix method for two scripts that download Twitter timelines. Unfortunately, it appears that the scripts are not run in parallel. Each of my scripts builds a separate MySQL table and I can see that the table for the second script only starts building after I interrupt the first script. Is this the case that @AashishP mentioned and should I be using threads? – user3722736 Jun 13 '14 at 14:00
  • Can anyone help me modify this to work in Windows (10)? I'm currently doing this in my Command Prompt window: C:/Python27/Scripts>pip NewCode5.1.py; pip NewCode5.2.py I've put these two .py files in the 'Scripts' folder, which is also where I've previously had to be to use the pip command. I get the following error: 'ERROR: unknown command NewCode5.1.py; ' Any suggestions? – LinnK Sep 25 '15 at 06:57

3 Answers3

1

As wanderlust mentioned, why do you want to do it this way and not via linux command line?

Otherwise, the solution you post is doing what it is meant to, i.e, you are doing this at the command line:

screen ./thread1.py
screen ./thread2.py

This will open a screen session and run the program and output within this screen session, such that you will not see the output on your terminal directly. To trouble shoot your output, just execute the scripts without the screen call:

import subprocess

subprocess.Popen(['./thread1.py'])
subprocess.Popen(['./thread2.py'])

Content of thread1.py:

#!/usr/bin/env python
def countToTen():
  for i in range(10):
  print i

countToTen()

Content of thread2.py:

#!/usr/bin/env python
def countToHundreds():
for i in range(10):
  print i*100

countToHundreds()

Then don't forget to do this on the command line:

chmod u+x thread*.py
jonnybazookatone
  • 2,188
  • 15
  • 21
  • Thanks, I've tried using the above script (in IDLE) but I got many errors. Traceback (most recent call last): line 3, in subprocess.Popen(['./thread1.py']) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 709, in __init__ errread, errwrite) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1326, in _execute_child raise child_exception OSError: [Errno 13] Permission denied – user3722736 Jun 11 '14 at 13:56
  • This is most likely related to the content of your 'thread1' and 'thread2'. Can you give any details? – jonnybazookatone Jun 11 '14 at 14:02
  • Sure, I got the UNIX solution to work but I'm curious what is wrong with my subprocess.Popen. Here is thread1: 'def countToTen(): for i in range(10): print i countToTen()' and thread2:'def countHundreds(): for i in range (10): print i*100 countHundreds()' I'm getting errors when I run this code: 'import subprocess subprocess.Popen(['thread1.py']) subprocess.Popen(['thread2.py'])' P.S Sorry for my bad formatting, I need to learn how to format here – user3722736 Jun 11 '14 at 15:22
  • You are missing the './' infront of thread1.py and thread2.py. Also, within each of the files, thread1.py and thread2.py you will need to point to the python executable, i.e. '#!/usr/bin/env python'. You also need to make them executable, i.e., 'chmod u+x thread*.py'. Otherwise it should work as you want. I will edit my post to include this. – jonnybazookatone Jun 11 '14 at 15:34
  • Thank you, it works now from command line (but not if I just try to run it from IDLE but perhaps you can't run concurrent programs there). I need to read up on the "chmod" line but this was a very valuable lesson for a Python beginner. – user3722736 Jun 11 '14 at 17:46
1

Use supervisord

supervisord is process control system just for the purpose of running multiple command line scripts.

It features:

  • multiple controlled processes
  • autorestarting failed runs
  • log stdout and stderr output
  • starting scripts in order (using priority)
  • command line utility to view latest log output, stop, start, restart the processes

This solution works only on *nix based systems, it is not available on Windows.

Jan Vlcinsky
  • 42,725
  • 12
  • 101
  • 98
1

You can also just open several Command Prompt windows to run several Python programs at once - just run one in each of them:

In each Command Prompt window, go to the correct directory (such as C:/Python27) and then type 'python YourCodeNo1.py' in one Command Prompt window, 'python YourCodeNo2.py' in the next one ect. .

I'm currently running 3 codes at one time in this way, without slowing any of them down.

LinnK
  • 385
  • 2
  • 4
  • 17