What am I doing wrong with sendline() in my twisted server?

Question

I'm trying to send the contents of a file over to the client 1 line at a time so that the client (written in Objective-C) can process each line individually. However the log of the client shows that the data being send over from the server is all coming through as 1 line and is apparently too large so the it cuts off mid way through one line causing the client to crash because of the unexpected syntax.

Is there something i'm doing on the server(written in python with twisted) that is causing the lines to not be sent separately?

Here is the particular code in the server that is holding me up at the moment.

def sendLine(self, line): 
    self.transport.write(line + '\r\n') 

def updateShiftList(self):

    #open the datesRequested file for the appropriate store and load the dates into a list
    fob = open('stores/'+self.storeName+'/requests/datesRequested','r')
    DATES_REQUESTED = fob.read()
    datesRequested = DATES_REQUESTED.split('\n')
    #open each date file that is listed in datesRequested
    for date in datesRequested:
        if os.path.isfile('stores/'+self.storeName+'/requests/' + date):
            fob2 = open('stores/'+self.storeName+'/requests/' + date,'r')
            #load the file into memory and split the individual requests up
            THE_REQUESTS = fob2.read()
            thedaysRequests = THE_REQUESTS.split('\n')
            for oneRequest in thedaysRequests:
                if len(oneRequest) > 4:
                    print "*)[*_-b4.New_REQUEST:"+oneRequest
                    self.sendLine('*)[*_-b4.New_REQUEST:'+oneRequest)
            fob2.close()
    fob.close()

So frustrating and i'm sure it's something easy. Thanks.

Thanks DSM, sendline wasn't being used only because I had been trying other methods to see if it changed the output. I updated the code to reflect this and added the proper closing on the files. Thanks. The lines are still being merged together even with another line that is in another method that is initiated prior to this method. — Guy Dayhoff, Mar 15 '12 at 13:56

score 0 · Accepted Answer · edited May 23 '17 at 11:48

This question concerns a topic that is frequently raised. There are a number of questions on stackoverflow about the same issue:

etc.

TCP sends an ordered, reliable stream of data.

Ordering means that bytes sent first arrive first.
Reliability means bytes sent will be delivered, or the connection will break. Data is not silently dropped.

Streaming is what mostly concerns your question. A stream is not divided up into separate messages. It consists of bytes, and the "boundary" between those bytes can move arbitrarily.

If you send "hello, " and then you send "world", the boundary between those two strings in the stream may disappear. The peer may receive "hello, world", or "h", "ello, world", or "he", "ll", "o,", " w", "or", "ld".

This is the very reason people use line-oriented protocols. The "\r\n" at the end of each logical message lets the receiver buffer and split the stream up into those original logical messages.

For a deeper dive, I recommend this video of a recent PyCon presentation: http://pyvideo.org/speaker/417/glyph

This all points towards the other end of your connection, the ObjC client, as the source of your misbehavior.

I want to say thank you. I've sort of come to this conclusion (the client being the issue) by playing with the code, but it's hard to know the proper question to ask when you're not 100% sure how to do what you're trying to do. You really helped me out here and gave me some good resources. So again thank you! — Guy Dayhoff, Mar 16 '12 at 13:07
The one issue I'm dealing with now on the side of the client is a break in the stream at 1448 bytes, i read that this is a normal breaking point for some NICs or something like that and it looks like your answer has clarified my lingering thoughts on what could be happening. So I will be working on my client side buffer knowing that the stream is ordered and reliable thanks and thanks and thanks again. — Guy Dayhoff, Mar 16 '12 at 13:07

What am I doing wrong with sendline() in my twisted server?

1 Answers1