struct.error: bad char in struct format

Question

I have a problem with a little server-client assignment in python 2.7.

The client can send 5 types of requests to the server:

1. get the server's ip
2. get contents of a directory on the server
3. run cmd command on the server and get the output
4. open a calculator on the server
5. disconnect

basically, this is the error I get:

line 19, in server
    data_size = calcsize(client_structs) - 3

struct.error: bad char in struct format

Would appreciate explanation about this error + how to solve it.

server code:

__author__ = 'eyal'

from struct import pack, unpack, calcsize
import socket
from os import listdir
from subprocess import check_output, call


def server():
    ser_soc = socket.socket()
    ser_soc.bind(("0.0.0.0", 8080))
    ser_soc.listen(1)
    while True:
        accept_flag = raw_input("Would you like to wait for a client? (y/n) ")
        if accept_flag == "y":
            client_soc, client_address = ser_soc.accept()
            while True:
                client_structs = client_soc.recv(1024)
                data_size = calcsize(client_structs) - 3
                data_str = 'c' * data_size
                unpacked_data = unpack("BH" + data_str, client_structs)
                if unpacked_data[0] == 1:
                    ip = socket.gethostbyname(socket.gethostname())
                    ip_data = 'c' * len(ip)
                    to_send = pack("BH" + str(len(ip)) + ip_data, unpacked_data[0], len(ip), ip)
                elif unpacked_data[0] == 2:
                    content = listdir(str(unpacked_data[2]))
                    content_str = "\r\n".join(content)
                    content_data = 'c' * len(content_str)
                    to_send = pack("BH" + str(len(content_str)) + content_data, unpacked_data[0],
                                   len(content_str), content_str)
                elif unpacked_data[0] == 3:
                    command = str(unpacked_data[2:]).split()
                    output = check_output(command)
                    message_data = 'c' * len(output)
                    to_send = pack("BH" + message_data, unpacked_data[0], len(output), output)
                elif unpacked_data[0] == 4:
                    call("gnome-calculator")
                    msg_data = 'c' * len("The calculator is open.")
                    to_send = pack("BH" + msg_data, unpacked_data[0], len("The calculator is open."),
                                   "The calculator is open.")
                elif unpacked_data[0] == 5:
                    client_soc.close()
                    break
                else:
                    to_send = pack("BH" + 'c' * len("invalid message type, try again"),
                                   unpacked_data[0], len("invalid message type, try again"),
                                   "invalid message type, try again")
                if unpacked_data[0] != 5:
                    client_soc.send(to_send)
        else:
            break
    ser_soc.close()


def main():
    server()


if __name__ == "__main__":
    main()

client code:

__author__ = 'eyal'


from struct import pack, unpack, calcsize
import socket


def client():
    my_soc = socket.socket()
    my_soc.connect(("127.0.0.1", 8080))
    while True:
        send_flag = raw_input("Would you like to send the server a request? (y/n) ")
        if send_flag == "y":
            msg_code = input("What type of request would you like to send?\n"
                             "1. Get the server's IP address.\n"
                             "2. Get content of a directory on the server.\n"
                             "3. Run a terminal command on the server and get the output.\n"
                             "4. Open a calculator on the server.\n"
                             "5. Disconnect from the server.\n"
                             "Your choice: ")
            if msg_code == 1 or msg_code == 4 or msg_code == 5:
                to_send = pack("BH", msg_code, 0)
            elif msg_code == 2:
                path = raw_input("Enter path of wanted directory to get content of: ")
                to_send = pack("BH" + 'c' * len(path), msg_code, len(path), path)
            elif msg_code == 3:
                command = raw_input("Enter the wanted terminal command, including arguments: ")
                to_send = pack("BH" + 'c' * len(command), msg_code, len(command), command)
            else:
                print "Invalid message code, try again\n"

            if 1 <= msg_code <= 5:
                my_soc.send(to_send)
        else:
            break
    data = my_soc.recv(1024)
    unpacked_data = unpack("BH" + 'c' * (calcsize(data) - 3), data)
    print "The server's response to your type-" + str(msg_code) + " request:"
    print unpacked_data[2]
    my_soc.close()


def main():
    client()


if __name__ == "__main__":
    main()

Answer 1

I think the problem is this:

data_size = calcsize(client_structs) - 3

The data the client sends over appears to be arbitrary binary data, not a struct format string. You can't call calcsize on that.

For example, if I select 1 on the client, it will pack the message code, 1, as a byte, and the number 0 as a short, so it'll send the b'\x01\x00\x00'. But on the server, you receive that and try to use it as a struct format. So, you get an error saying that b'\x01' is not a valid format code.

Since you've given us a ton of non-working code with no explanation of how it's supposed to work, it's hard to guess what you should be doing here, only that, whatever it is you want, this can't be the way to do it.

It looks like your format is always a 1-byte code, a 2-byte length, and then a bunch of arbitrary characters matching that length. If so, the way to parse that would be something like this:

code, length = unpack('BH', buffer[:3])
other_stuff = unpack('c' * length, buffer[3:])

Although really, there's not much point to using pack and unpack to pack a string of bytes; you're just going to get back the same thing that was already in buffer[3:], and on the other side you're just packing the path or whatever other thing it is into itself.

Anyway, again, this is just a guess, because I don't actually know how you expected your code to work, so this may not be what you wanted it to do.

---

Meanwhile, you've got at least one other serious problem that you need to fix. TCP sockets are byte streams, not message streams. When you do a send from the client, it may show up split across two recvs on the server, or merged with a previous send. You can't just recv and assume you have an entire message.

The worst thing about this problem is that when you're testing with localhost sockets on a machine that's not under load, it "works" 99.9999% of the time, meaning you may not realize you have a problem. But then, as soon as you try to deploy it on the internet or run it when your machine is busy, it starts failing all over the place, and only then do you realize that to debug the problem you have to write a huge chunk of code, and often restructure your whole program.

You seem to have designed a protocol that contains enough information to split the messages off the stream. But you have to actually do that, not just expect it to happen like magic.