0

I'm at the point in my network learning where I have just realized things aren't perfect and network packets can get out of sync and might not send everything in one packet.

My program works for small data that is being sent over the socket, but not a large file. I keep getting ValueError: Expecting ',' or Expecting ':' whenever trying to send a big file and I think it's because my receiving function is not that good. I've read up a lot on how you have to implement your own protocol for reading information from a socket but I still have found no information on how actually to implement that.

My program serializes some information on a computer and sends that over JSON to another computer. I heard somewhere that JSON automatically deals with sending packet information or something like that, but I'm not too clear about what that meant.

In any case, this is my receiving function:

def recieve(node, s, timeout=2):
    s.setblocking(False)

    #array of all data
    all_data = []

    begin = time.time()
    while True:
        #If data was received, break after timeout
        if all_data and time.time()-begin > timeout:
            break

        #If no data received at all, wait some more
        elif time.time() - begin > timeout * 2:
            break

        try:
            data = ''
            while len(data) < BUFFER_SIZE:
                data += s.recv(BUFFER_SIZE - len(data))
            if data:
                all_data.append(data.decode('utf-8'))
                #Reset begin for timeout
                begin = time.time()
        except:
            pass

    return json.loads(''.join(all_data))

And I just use `sendall(dumps((data1, data2, data3, data4, data5)).encode()) to encode my python information (which could be dictionaries, lists, etc).

I'm not sure if my timeouts even make any sense, I'm a complete beginner to network programming, and I'm not sure where to go from here. Should I use socket.makefile()? Do I still need to implement a network protocol when using JSON? How do I implement a network wrapper system if I need it?

Thanks so much in advance!

Ulrich Eckhardt
  • 16,572
  • 3
  • 28
  • 55
pradyuman
  • 1,089
  • 2
  • 14
  • 20
  • you should probably use `traceback.print_exc()` in your exceptions ... my guess is you are getting into your exception and loosing parts ... you should be able to stream arbitrarily large packets ... (and you should not have to worry about packet ordering or anything ... that should happen for free).. or maybe your timeout is not long enough ... I would have a timeout that is reset everytime you recieve anything – Joran Beasley Jul 14 '15 at 16:57
  • 1
    UDP or TCP? From what I can tell, your problem is that you're sending consecutive JSON chunks without delimiters or size headers, so you don't know when to stop reading. – nneonneo Jul 14 '15 at 16:57
  • 1
    For clarification, JSON is a data format, not really a protocol. – Rob Foley Jul 14 '15 at 16:58
  • 1
    1) you can't decode utf-8 in chunks, it is multibyte encoding 2) order of packets should not be your trouble if it is TCP – Andrey Jul 14 '15 at 17:00
  • To clarify, I'm using TCP. How do I combine multiple bytes and decode at the end? Do I have to do `b'.join(all_data).decode('utf-8')`? – pradyuman Jul 14 '15 at 17:43
  • @asuna yes, you can decode utf8 only if you have full message. if you use TCP then your app will receive full correctly ordered data or error (connection drop or whatever). – Andrey Jul 14 '15 at 17:59
  • @asuna, how are you marking the end of the message? Do you close() the connection on the client side? Do you send a special marker between JSON documents? – Robᵩ Jul 14 '15 at 18:27
  • I just check to see if any more data is coming in using the timeout. I haven't implemented any markers or anything. – pradyuman Jul 14 '15 at 18:39
  • 1
    I wouldn't use low-level TCP (well, not again, I have to admit) but a packet transfer like ZeroMQ, which takes a packet at the sender side and assembles the full packet at the receiving end again. Lots of complexity is removed from your code that way. – Ulrich Eckhardt Jul 14 '15 at 18:55
  • See also: http://stackoverflow.com/questions/21553327/why-is-except-pass-a-bad-programming-practice. Maybe Python is desperately trying to tell you something's wrong and you utterly ignore it? – Ulrich Eckhardt Jul 15 '15 at 19:09
  • I decided to use ZeroMQ and everything is working perfectly now. Thanks! – pradyuman Jul 16 '15 at 15:00

0 Answers0