Python serial fails reading large amounts of data

Question

So, I've got a microcontroller hooked up to my laptop using UART. In order to get a decent idea on when the microcontroller sends data to another microcontroller, I'm sending a copy of the data over the UART link, and timestamping it so that I can do some throughput and rough delay analysis later.

As it stands, I can transmit a series of 10 messages, and I'm able to read all the data 40% of the time. When I increase the amount of messages sent however, things get really wonky, and my script loses loads of data.

I'll warn you that it's been a while since I did python, so most of my code is very, lets say "un-pythonic".

The messages sent over UART are either data messages "sent: X" where X is 20bytes of garbage. Now, as this is not working as it should, the code is messy, and runs forever unless the magical send_done message is received. As I mentioned, this runs fairly well for a burst of 10 messages. From those, I can gather that the data rate is roughly 32kbps.

My question is whether it's possible to get more accuracy by fixing some obvious mistake that I cannot spot, or if it's simply python being too slow, and me needing to head over to C.

def main():
    launch_time=time.time()
    time.clock()
    [wday, month, day, clocktime, year]=time.ctime(launch_time).split(' ')  
    print(launch_time)

    milis=launch_time-math.floor(launch_time)
    milis=str(milis)
    print(milis)
    launch_time=clocktime+milis[1:len(clocktime)+1]

    Port5=serial.Serial(4,38300,rtscts=True,xonxoff=True) #Devkit

    sio5=io.TextIOWrapper(io.BufferedRWPair(Port5,Port5))
    timers=[]
    k=0
    msg=0
    buf=''
    while True:
        data=Port5.read(Port5.inWaiting())
        if data:
            if ":" in data: #Only safe as long as messages cannot contain lowercase s
                timers.append(time.clock())
                k+=1
                print(k)
            buf+=(data)
            #os.system('cls')
            #print(buf)
            #print(data)

        if "send_done" in buf:
            f=open("log.txt","a")
            f.write(launch_time+'\n')
            f.close()
            #print(len(timers))
            #print(timers)
            for string in buf.split("sent:"):
                if "send_done" in string:
                    [string, discard]=string.split("send_done")

                if string:
                    print(msg)
                    print(string)
                    print(k)
                    f=open("log.txt","a")
                    f.write(addseconds(launch_time,str(timers[msg])[0:12])+' '+string.encode('Hex').upper()+'\n')
                    f.close()

                    msg+=1

            print(timers)
            return  

Edit It seems like this may not be an issue on this end of the transmission, as after testing the threaded script discussed in A similar post, as well as testing with Termite, the full set of data does not arrive 100% of the time on either. I may have to tinker with some more settings on the micro controller.

Answer 1

Are you sure it's 38300 and not 38400 baud?

There should be no performance bottleneck below 4 KB/s of data stream even for python on the worst current PCs (and 38.3kbps is around that). To test that, you should write directly to file every data read from the serial port, without reopening it, printing to console or whatever.

Running 'cls' as system command and printing an ever incrementing size buffer to the screen is very slow on the other hand, that may cause problems for you.

Also please note that buf += (data) (why not buf += data without the parentheses?) will build a new string every round so if it grows huge, memory allocation + data copy will also be a performance problem, but not after 10 messages.

EDIT: A possible (but in your case most probably unnecessary) performance improvement could be to store the incoming nonempty data chunks in a list. Appending to the end of a list will not cost more as the list size grows. Then join only the last 9 elements of it to check for the 'send_data' string. Something like this:

incoming = ['asdf', None, 'vf', None, None, 'afsd', 'gfts', 'end', None, '_', 'do', 'nett', 'rest of data not processed']
in_list = []

for data in incoming:
    if data:
        in_list.append(data)
        if "send_done" in "".join(in_list[-9:]):
            string, discard = "".join(in_list).split("send_done")
            print "Str: '%s' discard: '%s'" % (string, discard)
            break