Pickle EOFError: Ran out of input when recv from a socket

Question

I am running a very simple python (3.x) client-server program (both locally on my PC) for a school project (not intended for the real world) which just sends messages back-and-forth (like view customers, add customer, delete customer, etc... real basic).

Sometimes the data can be multiple records which I had stored as namedTuples (just made sense) and then went down the path of using Pickle to transfer then.

So for example on the client I do something like this:

s.send(message.encode('utf-8'))
pickledResponse = s.recv(4096);
response = pickle.loads(pickledResponse)

Now ever so often I get the following error:

response = pickle.loads(pickledResponse)
EOFError: Ran out of input

My fear is that this has something to do with my socket (TCP) transfer and maybe somehow I am not getting all the data in time for my pickle.loads - make sense? If not I am really lost as to why this would be happening so inconsistently.

However, even if I am right I am not sure how to fix it (quickly), I was considering dropping pickle and just using strings (but couldn't this suffer from the same fate)? Does anyone have any suggestions?

Really my message are pretty basic - usually just a command and some small data like "1=John" which means command (1) which is FIND command and then "John" and it returns the record (name, age, etc...) of John (as a namedTuple - but honestly this isn't mandatory).

Any suggestions or help would be much appreciated, looking for a quick fix...

Answer 1

The problem with your code is that recv(4096), when used on a TCP socket, might return different amount of data from what you might have expected, as they are sliced at packet boundaries.

The easy solution is to prefix each message with length; for sending like

import struct
packet = pickle.dumps(foo)
length = struct.pack('!I', len(packet)
packet = length + packet

then for receiving

import struct

buf = b''
while len(buf) < 4:
    buf += socket.recv(4 - len(buf))

length = struct.unpack('!I', buf)[0]
# now recv until at least length bytes are received,
# then slice length first bytes and decode.

---

However, Python standard library already has a support for message oriented pickling socket, namely multiprocessing.Connection, that supports sending and receiving pickles with ease using the Connection.send and Connection.recv respectively.

Thus you can code your server as

from multiprocessing.connection import Listener

PORT = 1234
server_sock = Listener(('localhost', PORT))
conn = server_sock.accept()

unpickled_data = conn.recv()

and client as

from multiprocessing.connection import Client

client = Client(('localhost', 1234))
client.send(['hello', 'world'])

Answer 2

For receiving everything the server sends until it closes its side of the connection try this:

import json
import socket
from functools import partial


def main():
    message = 'Test'

    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
        sock.connect(('127.0.0.1', 9999))

        sock.sendall(message.encode('utf-8'))
        sock.shutdown(socket.SHUT_WR)

        json_response = b''.join(iter(partial(sock.recv, 4096), b''))

    response = json.loads(json_response.decode('utf-8'))
    print(response)


if __name__ == '__main__':
    main()

I've used sendall() because send() has the same ”problem” as recv(): It's not guaranteed everything is sent. send() returns the number of bytes actually sent, and the programmer has to make sure that matches the length of the argument and if not to send the rest until everything is out. After sending the writing side of the connection is closed (shutdown()) so the server knows there is no more data coming from the client. After that, all data from the server is received until the server closes its side of the connection, resulting in the empty bytes object returned from the recv() call.

Here is a suitable socketserver.TCPServer for the client:

import json
from socketserver import StreamRequestHandler, TCPServer


class Handler(StreamRequestHandler):

    def handle(self):
        print('Handle request...')
        message = self.rfile.read().decode('utf-8')
        print('Received message:', message)
        self.wfile.write(
            json.dumps(
                {'name': 'John', 'age': 42, 'message': message}
            ).encode('utf-8')
        )
        print('Finished request.')



def main():
    address = ('127.0.0.1', 9999)
    try:
        print('Start server at', address, '...')
        server = TCPServer(address, Handler)
        server.serve_forever()
    except KeyboardInterrupt:
        print('Stopping server...')


if __name__ == '__main__':
    main()

It reads the complete data from the client and puts it into a JSON encoded response with some other, fixed items. Instead of the low level socket operations it makes use of the more convenient file like objects the TCPServer offers for reading and writing from/to the connection. The connection is closed by the TCPServer after the handle() method finished.