0

I want to run a function continuoulsy in parallel to my main process.How do i do it in python?multiprocessing?threading or thread module? I am new to python.Any help much appreciated.

androidnewbie
  • 423
  • 2
  • 8
  • 15
  • 4
    Depends on what you function should do. Can you pls provide more detailed description? – Alexey Kachayev Dec 18 '12 at 12:55
  • 1
    `threading` (in Cpython) will (probably) share resources on 1 core of your machine whereas `multiprocessing` will spawn a separate process (which allows the OS to delegate that to a different core of your machine). – mgilson Dec 18 '12 at 13:05
  • @ Alexey Kachayev i have some python scripts ,and i need to monitor the stderr continuosly in parallel to main script,and take an action if a something gets logged into stderr.The action would involve accessing an object created by the main script and notifying it of the event. – androidnewbie Dec 18 '12 at 13:10
  • i voted to close this question because both the question/title ask about python threading... but then you wrote in the comments that ACTUALLY you need to do something with stderr... so either the question should be closed or updated. – Trevor Boyd Smith Nov 06 '17 at 18:49

4 Answers4

2

If the aim is to capture stderr and do some action you can simply replace sys.stderr by a custom object:

>>> import sys
>>> class MyLogger(object):
...     def __init__(self, callback):
...             self._callback = callback
...     def write(self, text):
...             if 'log' in text:
...                     self._callback(text)
...             sys.__stderr__.write(text)   # continue writing to normal stderr
... 
>>> def the_callback(s):
...     print('Stderr: %r' % s)
... 
>>> sys.stderr = MyLogger(the_callback)
>>> sys.stderr.write('Some log message\n')
Stderr: 'Some log message'
Some log message
>>> 
>>> sys.stderr.write('Another message\n')
Another message

If you want to handle tracebacks and exceptions you can use sys.excepthook.

If you want to capture logs created by the logging module you can implement your own Handler class similar to the above Logger but reimplementing the emit method.

A more interesting, but less practical solution would be to use some kind of scheduler and generators to simulate parallel execution without actually creating threads(searching on the internet will yield some nice results about this)

Bakuriu
  • 98,325
  • 22
  • 197
  • 231
0

It definitely depends on your aim, but I'd suggest looking at the threading module. There are many good StackOverflow questions on the use of threading and multithreading (e.g., Multiprocessing vs Threading Python).

Here's a brief skeleton from one of my projects:

import threading  # Threading module itself 
import Queue      # A handy way to pass tasks to your thread 

job_queue = Queue.Queue()
job_queue.append('one job to do')

# This is the function that we want to keep running while our program does its thing
def function_to_run_in_background():
    # Do something...here is one form of flow control 
    while True:
        job_to_do = job_queue.get()  # Get the task from the Queue
        print job_to_do              # Print what it was we fetched 
        job_queue.task_done()        # Signal that we've finished with that queue item

# Launch the thread...
t = threadingThread(target=function_to_run_in_background, args=(args_to_pass,)) 
t.daemon = True  # YOU MAY NOT WANT THIS: Only use this line if you want the program to exit without waiting for the thread to finish 
t.start()        # Starts the thread 
t.setName('threadName') # Makes it easier to interact with the thread later 

# Do other stuff 
sleep(5)
print "I am still here..."
job_queue.append('Here is another job for the thread...')

# Wait for everything in job_queue to finish. Since the thread is a daemon, the program will now exit, killing the thread. 
job_queue.join()
Community
  • 1
  • 1
Chris
  • 21
  • 1
  • if the thread function_to_run_in_background() is non-daemonic,then will the thread die if my main process throws exceptions and exits? – androidnewbie Dec 21 '12 at 06:08
  • Actually what i need is 1) to run a process in parallel to my main process 2)the parallel process should not exit if the main process exits or terminates because of any unhandled exception – androidnewbie Dec 21 '12 at 11:17
  • @androidnewbie Interesting...my suggestion should meet condition 1 but _not_ condition 2. If you did spawn such a process, wouldn't it be nearly impossible to kill it (you'd have to search running processes for it)? Could you turn the solution around, making the logging process the main thread and then spawning the work in another thread? Then, the working thread could die, leaving the main logging thread to run, log the error, take action, and possibly even launch another worker thread. – Chris Jan 03 '13 at 14:32
0

if you just want to run a function in background in the same process, do:

import thread

def function(a):
    pass

thread.start_new(function, (1,)) # a is 1 then
User
  • 14,131
  • 2
  • 40
  • 59
0

I found that client-server architecture was solution for me. Running server, and spawning many clients talking to server and between clients directly, something like messenger.

Talking/comunication can be achieved through network or text file located in memory, (to speed things up and save hard drive).

Bakuriu: give u a good tip about logging module.

okobaka
  • 576
  • 4
  • 8