Why does my socket connection between two python scripts break if one of them is launched with Popen?

Question

So I have two very simple python scripts that communicate over a socket. Right now they are both running on the same windows PC.

Here's controller.py:

import socket
import time
import sys
from subprocess import Popen, CREATE_NEW_CONSOLE

HOST = '192.168.1.107'          # The remote host
PORT = 50557                        # The same port as used by the server
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

try:
    s.connect((HOST, PORT))
except:
    Popen([sys.executable, 'driver.py'], creationflags=CREATE_NEW_CONSOLE)
    time.sleep(0.2);
    s.connect((HOST, PORT))

s.send(sys.argv[1])
data = s.recv(1024)

s.close()
print 'Received', repr(data)

And here's driver.py:

import socket

HOST = ''                                # Symbolic name meaning the local host
PORT = 50557                    # Arbitrary non-privileged port

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)

while 1:
    print 'Waiting for controller connection.'
    conn, addr = s.accept()
    print 'Connected to ', addr
    while 1:
        data = conn.recv(1024)
        print 'Recieved ', data
        if not data: break
        if data == 'Q': quit()
        print 'A.'      
        conn.send(data[::-1])
        print 'B.'
    print 'C.'

conn.close()        

If I open two cmd windows and python <filename>.py <arg> them both everything works fine. I can leave driver.py running and run controller.py over and over again. Until I kill the driver by sending a "Q".

The try/except statement opens up a new window and runs driver.py if the connection can't be made. Theoretically this just makes sure that the receiver is running before it sends anything. This ALMOST works but for some reason driver.py hangs inside the second while loop for reasons I cannot explain. Here's the output from sequential controller.py calls:

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

>python controller.py FirstMsg
Received 'gsMtsriF'

>python controller.py SecondMsg
Received 'gsMdnoceS'

>python controller.py Q
Received ''

>python controller.py ThirdMsg
Received 'gsMdrihT'

>python controller.py FourthMsg

(After FourthMsg controller.py hangs forever)

Here's the output from driver.py

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

>python driver.py
Waiting for controller connection.
Connected to  ('192.168.1.107', 49915)
Recieved  FirstMsg
A.
B.
Recieved
C.
Waiting for controller connection.
Connected to  ('192.168.1.107', 49916)
Recieved  SecondMsg
A.
B.
Recieved
C.
Waiting for controller connection.
Connected to  ('192.168.1.107', 49917)
Recieved  Q

(Whereupon it returns to the regular command prompt.) The new window created with Popen contains this: Waiting for controller connection. Connected to ('192.168.1.107', 49918) Recieved ThirdMsg A. B.

So in the weird new window (which I just noticed does not look or act like a regular cmd window) the program seems to hang on data = conn.recv(1024).

Why does driver.py act differently in the new window, and how can I fix it?

Answer 1

You asked for sockets and Popen, my answer will ignore this and will try to propose solution for the use case, you have shown - remote communication over network.

With sockets it is very easy to run into never ending problems. If you have both endpoints under your control, zeromq communication will turn your programming back to having fun.

You may either use zeromq directly (if you check examples for pyzmq, you will find they are really short and easy while serving very well in many difficult scenarios), but today I will show use of zerorpc library, which makes remote calls even simpler.

Install zerorpc

$ pip install zerorpc

Write your worker.py

def reverse(text):
    """ return reversed argument """
    return text[::-1]

def makebig(text):
    """ turn text to uppercase letters """
    return text.upper()

Call worker methods from command line

Start server serving worker.py functions

$ zerorpc --server --bind tcp://*:5555 worker
binding to "tcp://*:5555"
serving "worker"

Consume the services (from command line)

Simple call, reporting available functions to call

$ zerorpc tcp://localhost:5555
connecting to "tcp://localhost:5555"
makebig turn text to uppercase letters 
reverse return reversed argument 

Then call these functions:

$ zerorpc tcp://localhost:5555 reverse "alfabeta"
connecting to "tcp://localhost:5555"
'atebafla'

$ zerorpc tcp://localhost:5555 makebig "alfabeta"
connecting to "tcp://localhost:5555"
'ALFABETA'

Note: so far we have wrote 7 lines of code and are already able calling it remotely.

Using from Python code

Command line utility zerorpc is just handy utility, but you are free to integrate without it, using pure Python.

Consuming from python code client.py

import zerorpc

client = zerorpc.Client()
url = "tcp://localhost:5555"
client.connect(url)

print client.reverse("alfabeta")
print client.makebig("alfabeta")

and run it

$ python client.py
atebafla
ALFABETA

Run remote server from python code server.py

import zerorpc
import worker

url = "tcp://*:5555"

srv = zerorpc.Server(worker)
srv.bind(url)
srv.run()

Stop older server started by command zerorpc, if it still runs.

Then start out python version of the server:

$ python server.py

(it does not print anything, just waits to serve).

Finally, consume it from your Python code

$ python client.py 
atebafla
ALFABETA

Conclusions

• Using zeromq or zerorpc simplifies the situation a lot
• Play with starting client.py and server.py in various orders
• try to run multiple clients
• try to consume services from another computer (over network)
• zerorpc provides more patterns (streamer, publish/subscribe), see zerorpc test suite
• zerorpc solves the integration task and let you focus on coding real functions