-1

I have two apps sending tcp packages, both written in python 2. When client sends tcp packets to server too fast, the packets get concatenated. Is there a way to make python recover only last sent package from socket? I will be sending files with it, so I cannot just use some character as packet terminator, because I don't know the content of the file.

fulaphex
  • 2,879
  • 3
  • 19
  • 26
  • 2
    This is just not how TCP works. – David Schwartz Feb 15 '16 at 22:08
  • As many have suggested, the so called _length-prefixing_ is the way to go. I don't use Python, but I'm able to send files without any problem from my program - which uses length-prefixing for every "packet" or file part it sends. – Visual Vincent Feb 16 '16 at 18:34
  • I have just made all the packets the same size by adding padding and including file sizes before the actual file. – fulaphex Feb 16 '16 at 18:35

4 Answers4

3

TCP uses packets for transmission, but it is not exposed to the application. Instead, the TCP layer may decide how to break the data into packets, even fragments, and how to deliver them. Often, this happens because of the unterlying network topology.

From an application point of view, you should consider a TCP connection as a stream of octets, i.e. your data unit is the byte, not a packet.

If you want to transmit "packets", use a datagram-oriented protocol such as UDP (but beware, there are size limits for such packets, and with UDP you need to take care of retransmissions yourself), or wrap them manually. For example, you could always send the packet length first, then the payload, over TCP. On the other side, read the size first, then you know how many bytes need to follow (beware, you may need to read more than once to get everything, because of fragmentation). Here, TCP will take care of in-order delivery and retransmission, so this is easier.

Has QUIT--Anony-Mousse
  • 76,138
  • 12
  • 138
  • 194
  • I don't want to use UDP, because it loses packets and delivers them in random permutation. For file transfer I cannot let this happen – fulaphex Feb 15 '16 at 22:08
  • Then do the approach I outlined with framing the "packets" yourself by sending a length first. Because TCP is a stream of bytes, not of packets. – Has QUIT--Anony-Mousse Feb 15 '16 at 22:22
  • Trying to implement this method of Yours with giving the lenght. As You mentioned, I might read the whole file and calculate the size. It going through the whole file again. I wanted to use `sys.getfilesize()` function, but it gives me another result than calculating the bytes and doesn't work. – fulaphex Feb 15 '16 at 23:16
  • I got the last issue solved. The problem was, that I've been measuring the length of the object string and it has 37 bytes more than letters. – fulaphex Feb 16 '16 at 07:27
1

TCP is a streaming protocol, which doesn't expose individual packets. While reading from stream and getting packets might work in some configurations, it will break with even minor changes to operating system or networking hardware involved.

To resolve the issue, use a higher-level protocol to mark file boundaries. For example, you can prefix the file with its length in octets (bytes). Or, you can switch to a protocol that already handles this kind of stuff, like http.

user4815162342
  • 141,790
  • 18
  • 296
  • 355
0

First you need to know if the packet is combined before it is sent or after. Use wireshark to check it the sender is sending one packet or two. If it is sending one, then your fix is to call flush() after each write. I do not know the answer if the receiver is combining packets after receiving them.

You could change what you are sending. You could send bytes sent, followed by the bytes. Then the other side would know how many bytes to read.

Robert Jacobs
  • 3,266
  • 1
  • 20
  • 30
  • Do You mean I should add something to my message so I know where the end is? What if this same sequence of bytes occurs in a file I want to send? – fulaphex Feb 15 '16 at 22:01
  • That was my second answer. If you "KNOW" that you will send 4 bytes size, followed by that many bytes, repeat loop until 0 or EOF. Then those 4 bytes will always be interpreted as a size. Use struct to convert an integer to 4 bytes network order. The flush() answer doesn't require any change to your packet. – Robert Jacobs Feb 15 '16 at 22:04
  • 1
    Yeah, but I don't think, that TCP sockets in python have flush function implemented. – fulaphex Feb 15 '16 at 22:09
0

Normally, TCP_NODELAY prevents that. But there are very few situations where you need to switch that on. One of the few valid ones are telnet style applications.

What you need is a protocol on top of the tcp connection. Think of the TCP connection as a pipe. You put things in one end of the pipe and get them out of the other. You cannot just send a file through this without both ends being coordinated. You have recognised you don't know how big it is and where it ends. This is your problem. Protocols take care of this. You don't have a protocol and so what you're writing is never going to be robust.

You say you don't know the length. Get the length of the file and transmit that in a header, followed by the number of bytes.

For example, if the header is a 64bits which is the length, then when you receive your header at the server end, you read the 64bit number as the length and then keep reading until the end of the file which should be the length.

Of course, this is extremely simplistic but that's the basics of it.

In fact, you don't have to design your own protocol. You could go to the internet and use an existing protocol. Such as HTTP.

hookenz
  • 36,432
  • 45
  • 177
  • 286