Sockets Python 3.5: Socket server hangs forever on file receive

Question

I'm trying to write a Python program that can browse directories and grab files w/ sockets if the client connects to the server. The browsing part works fine, it prints out all directories of the client. Here's a part of the code:

with clientsocket:
    print('Connected to: ', addr)
    while True:
        m = input("Command > ")

        clientsocket.send(m.encode('utf-8'))
        data = clientsocket.recv(10000)

        if m == "exit":
            clientsocket.close()
        if m.split()[0] == 'get':
            inp = input("Filename > ")
            while True:
                rbuf = clientsocket.recv(8192)                
                if not rbuf:
                    break

                d = open(inp, "ab")
                d.write(rbuf)
                d.close()


        elif data.decode('utf-8').split()[0] == "LIST":
            print(data.decode('utf-8'))
        if not data:
            break

However, the problem lies in here:

if m.split()[0] == 'get':
    inp = input("Filename > ")
    while True:
        rbuf = clientsocket.recv(8192)                
        if not rbuf:
            break

It seems to be stuck in an infinite loop. What's more interesting is that the file I'm trying to receive is 88.3kb, but what the file returns is 87kb while it's in the loop, which is very close...

I tried receiving a python script at one time as well (without the loop) and it works fine.

Here's some of the client code:

while True:
    msg = s.recv(1024).decode('utf-8')


    if msg.split()[0] == "list":
        dirs = os.listdir(msg.split()[1])
        string = ''
        for dira in dirs:
            string += "LIST " + dira + "\n"
        s.send(string.encode('utf-8'))
    elif msg == "exit":
        break
    else:
        #bit that sends the file
        with open(msg.split()[1], 'rb') as r:
            s.sendall(r.read())

So my question is, why is it getting stuck in an infinite loop if I have it set up to close when there is no data, and how can I fix this?

I'm sort of new to network programming in general, so forgive me if I miss something obvious.

Thanks!

That was for debugging earlier, I guess I don't need it anymore. — Megalegacy, Aug 10 '16 at 21:45

score 2 · Accepted Answer · answered Aug 10 '16 at 22:01

I think I know what's the problem, but I may be wrong. It happened to me several times, that the entire message is not received in one recv call, even if I specify the correct length. However, you don't reach the end of stream, so your program keeps waiting for remaining of 8192 bytes which never arrives.

Try this:
Sending file:

#bit that sends the file
with open(msg.split()[1], 'rb') as r:
   data = r.read()
   # check data length in bytes and send it to client
   data_length = len(data)
   s.send(data_length.to_bytes(4, 'big'))
   s.send(data)
s.shutdown(socket.SHUT_RDWR)
s.close()

Receiving the file:

# check expected message length
remaining = int.from_bytes(clientsocket.recv(4), 'big')
d = open(inp, "wb")
while remaining:
    # until there are bytes left...
    # fetch remaining bytes or 4094 (whatever smaller)
    rbuf = clientsocket.recv(min(remaining, 4096))
    remaining -= len(rbuf)
    # write to file
    d.write(rbuf)
d.close()

how does `remaining = int.from_bytes(clientsocket.recv(4), 'big')` work? — Megalegacy, Aug 10 '16 at 22:11
`clientsocket.recv(4)` receive first four bytes which encodes data length, than, bytes are converted to int using big-endian bytes order, the same which we used to encode int to bytes. — warownia1, Aug 10 '16 at 22:17

score 1 · Answer 2 · answered Aug 10 '16 at 22:16

There are several issues with your code.

First:

clientsocket.send(m.encode('utf-8'))
data = clientsocket.recv(10000)

This causes the file to be partially loaded to data variable when you issue get statement. That's why you don't get full file.

Now this:

while True:
    rbuf = clientsocket.recv(8192)                
    if not rbuf:
        break
    ...

You indeed load full file but the client never closes the connection (it goes to s.recv() after sending the file) so if statement is never satisfied. Thus this loop gets blocked on the clientsocket.recv(8192) part after downloading the file.

So the problem is that you have to somehow notify the downloader that you've sent all the data even though the connection is still open. There are several ways to do that:

You calculate the size of the file and send it as a first few bytes. For example, say the content of the file is ala ma kota. These are 11 bytes and thus you send \x11ala ma kota. Now receiver knows that first byte is size and it will interpret it as such. Of course one byte header isn't much (you would only be able to send max 256 byte files) so normally you go for for example 4 bytes. So now your protocol between client and server is: first 4 bytes is the size of the file. Thus our initial file would be sent as \x0\x0\x0\x11ala ma kota. The drawback of this solution is that you have to know the size of the content before sending it.
You mark the end of the stream. So you pick a particular character, say X and you read the straem until you find X. If you do then you know that the other side sent everything it has. The drawback is that if X is in the content then you have to escape it, i.e. additional content processing (and interpretation on the other side) is needed.

Sockets Python 3.5: Socket server hangs forever on file receive

2 Answers2

Linked