Invalid argument error from background process when main script stops

Question

I have this code to get tweets by running a background process. The following script is run from main script using subprocess.Popen function. So that the main script will stop executing after invoking the background process script.

def start_listner(unique_id, keyword, limit=200):
    class CustomStreamListener(tweepy.StreamListener):

        def __init__(self, api):
            logger.info('runnning')
            self.api = api
            super(tweepy.StreamListener, self).__init__()

            #setup rabbitMQ Connection


        def on_status(self, status):
            print status.text.encode('utf-8'), "\n"
            #queue the tweet and writes the tweet to the log

        def on_error(self, status_code):
          #some code to not kill the stream

        def on_timeout(self):
          #some code to not kill the stream

    sapi = tweepy.streaming.Stream(auth, CustomStreamListener(api))
    try:
        logger.info('tracking started')
        logger.info(keyword)
        logger.info(type(keyword))
        kw = keyword
        sapi.filter(track=[kw])  # keeps listening to the streaming api
    except Exception, err:
        logger.info(kw) # fails at this place when main py stops
        logger.info(err)

if __name__ == "__main__":
    logger.info("just now started")
    try:
        a = str(sys.argv[1])
        b = str(sys.argv[2])
        #c = int(sys.argv[5])
        logger.info(a)
        logger.info(b)
    except Exception, err:
        logger.info("inside main")
    start_listner(a, b)

According to the highest voted answer here I use the following main script to invoke the StreamingAnalytics.py(above code)

import time
import subprocess
subprocess.Popen(["python", "StreamingAnalytics.py", 'SriLankaQ', 'lanka'])

print 'I could escape.........'
time.sleep( 15 )

I have added a sleep so the tweets will be successfully added to the RabbitMQ queue during that time. But as soon as the main script stops the background process prints the following error.

2015-12-22 16:28:16,559 - main - INFO - {'text': 'RT @Dory: lanka singing Hotline bling \xf0\x9f\x98\x82\xf0\x9f\x98\x82 'source': u'Twitter for iPhone'}

2015-12-22 16:28:17,752 - main - INFO - lanka

2015-12-22 16:28:17,752 - main - INFO - [Errno 22] Invalid argument

UPDATE: Since I thought its an issue in passing arguments I removed the use of arguments by writing them to a file by main script and reading the file from background process file. So,

subprocess.Popen(["python", "StreamingAnalytics.py"])

But still the same error comes. Using the traceback module I could print more information on this error.

2015-12-24 11:01:16,562 - __main__ - INFO - Traceback (most recent call last):
File "StreamingAnalytics.py", line 84, in <module>
    sapi.filter(track=[keyword])
File "C:\Python27\lib\site-packages\tweepy\streaming.py", line 445, in filter
    self._start(async)
File "C:\Python27\lib\site-packages\tweepy\streaming.py", line 361, in
_start
    self._run()
File "C:\Python27\lib\site-packages\tweepy\streaming.py", line 294, in _run
    raise exception IOError: [Errno 22] Invalid argument

So I had to modify a few things, but [my example](https://gist.github.com/waynew/c47fe03405e451709906) works just fine for me. Does that work for you, too? I have your first example launcher running it (i.e. `subprocess.Popen(["python", "StreamingAnalytics.py", 'SriLankaQ', 'lanka'])`) — Wayne Werner, Dec 24 '15 at 15:26
Can you have your `StreamingAnalytics.py` running by simply calling it directly without using the `subprocess.Popen`? (i.e. call `python StreamingAnalytics.py` in a terminial) — Jon, Dec 24 '15 at 23:47
@WayneWerner It works now! I have a print statement inside the `on_status` function When I removed that it worked. I included that also to my question. I think the problem is with that. Anyway its your answer that lead me to work this. If you add it as an answer with some explanation why it works when removes print. I would be happy to accept it as the correct answer. — Marlon Abeykoon, Dec 25 '15 at 05:05
Try using `multiprocessing.Process` instead of starting Python with a subprocess... — Roland Smith, Dec 30 '15 at 12:51
@MarlonAbeykoon It is a different process, so it should continue. — Roland Smith, Dec 30 '15 at 12:55
@MarlonAbeykoon so `tweetpy` is poorly coded and hides real error from you, please hack / instrument it to get real traceback. — Dima Tisnek, Dec 31 '15 at 10:10

score 2 · Answer 1 · answered Dec 30 '15 at 12:45

2

Your traceback is obscured by tweetpy.

My recommendation:

edit tweepy/streaming.py
find the two lines exception = ...
add logging.exception("foobar") right before or after
run again
post complete traceback

answered Dec 30 '15 at 12:45

Dima Tisnek

11,241
4
68
120

1

This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. - [From Review](/review/low-quality-posts/10727819) – Mad Physicist Dec 30 '15 at 15:59
@MadPhysicist I challenge you to post a better answer! Now if you wanna be all legalist, please read OP carefully, it doesn't contain a single question. – Dima Tisnek Dec 31 '15 at 10:09
1

Just posting an answer for the sake of saying you did it is pointless. While I completely agree that your approach is the right one, you are not posting an answer as such. The steps you recommend are basically a request for more information, which should go in a comment. Once the OP has complied, you can post an answer fixing his actual problem. – Mad Physicist Dec 31 '15 at 13:27
@qarma Can you tell me the exact line in https://github.com/tweepy/tweepy/blob/master/tweepy/streaming.py? BTW I have found out that `print` statement in background process script lead to this error (as helped by @Wayne in question comments) When I remove the print statement it worked. If you can extend your answer the reason behind this I'd be happy to accept this answer – Marlon Abeykoon Dec 31 '15 at 17:25

Invalid argument error from background process when main script stops

1 Answers1