String Concatenation Python 2.5 (using cStringIO)

Question

I'm using Google Appengine with Python 2.5 and I have a function that is causing a bottleneck. I pass it a list of 200 Model instances retrieved from the datastore, and it returns it in a json format which I then pass to the client.

I originally used += to concatenate all the values together, but it was taking around 30 seconds for the server to respond with the JSON. I ran some checks and the code before this function runs in under a second. It is the last statement before the server responds with the JSON and the time it takes to reach the client averages at 1 second (on my local network). This function takes on average 30 seconds to execute.

I read this article and tried using the cStringIO method (I also used the list join method but it took the same amount of time and cStringIO uses less memory so I stuck with it). However, this took around the same time as += concatenation (sometimes longer). Can anyone see any issues I have with my code that might make it slower?

EDIT: Boss says it has to be done this way. No json libraries (take it up with him).

EDIT 2: LastName Model:

class LastName(db.Model): 
    entry = db.ReferenceProperty(AlumniEntry, collection_name='last_names') 
    last_name = db.StringProperty(indexed=False) 
    last_name_search = db.StringProperty()

AlumniEntry is the Model that is queried. I pass the list that I get back from the ds to get_json_from_alumnus() (alumnus parameter).

def get_json_from_alumnus(alumnus, search, total=0):
    if len(alumnus) > 0:
        from cStringIO import StringIO
        concat_file = StringIO()

        concat_file.write('{ "alumnus": [')
        i = 0
        for alumni in alumnus:
            if alumni.author:
                author = alumni.author.nickname()
            else:
                author = 'Anonymous'

            concat_file.write('{ ')
            concat_file.write('"author": "')
            concat_file.write(author)
            concat_file.write('", ')
            concat_file.write('"title": "')
            concat_file.write(alumni.title)
            concat_file.write('", ')
            concat_file.write('"first_name": "')
            concat_file.write(alumni.first_name)
            concat_file.write('", ')

            concat_file.write(' "last_names": [')
            j = 0
            for lname in alumni.last_names:
                concat_file.write('{ "last_name": "')
                concat_file.write('lname.last_name')
                concat_file.write('" }')
                if not j == alumni.last_names.count() - 1:
                    #last_names += ','
                    concat_file.write(',')
                j +=1
            concat_file.write('], ')

            concat_file.write(' "addresses": [')
            j = 0
            for address in alumni.addresses:
                if address.street == '' and address.city == '' and address.state == '' and address.zip_code == '':
                    break

                concat_file.write('{ "address":{ "street" : "')
                concat_file.write(address.street)
                concat_file.write('", ')
                concat_file.write('"city" : "')
                concat_file.write(address.city)
                concat_file.write('", ')
                concat_file.write('"state" : "')
                concat_file.write(address.state)
                concat_file.write('", ')
                concat_file.write('"zip_code" : "')
                concat_file.write(address.zip_code)
                concat_file.write('" } }')

                if not j == alumni.addresses.count() - 1:
                    concat_file.write(',')
                j += 1
            concat_file.write('], ')

            concat_file.write(' "numbers": [')
            j = 0
            for phone_number in alumni.phone_numbers:
                concat_file.write('{ "phone_number": "')
                concat_file.write(phone_number.number)
                concat_file.write('" }')
                if not j == alumni.phone_numbers.count() - 1:
                    concat_file.write(',')
                j += 1
            concat_file.write('], ')

            concat_file.write(' "emails": [')
            j = 0
            for email in alumni.emails:
                concat_file.write('{ "email": "')
                concat_file.write(email.email)
                concat_file.write('" }')
                if not j == alumni.emails.count() - 1:
                    concat_file.write(',')
                j += 1
            concat_file.write('], ')

            concat_file.write('"grad_year": "')
            concat_file.write(alumni.grad_year)
            concat_file.write('", ')
            concat_file.write('"elementary": "')
            concat_file.write(alumni.elementary)
            concat_file.write('", ')
            concat_file.write('"entered": "')
            concat_file.write(str(alumni.entered.strftime('%B %d %Y')))
            concat_file.write('", ')
            concat_file.write('"key": "')
            concat_file.write(str(alumni.key()))
            concat_file.write('" ')
            concat_file.write('}')

            if not  i == len(alumnus) - 1:
                concat_file.write(',')
            i += 1
        concat_file.write('], "total" : "')
        concat_file.write(str(total))
        concat_file.write('" }')
    else:
        concat_file.write('{ "alumnus": "No Alumni Entered Yet!" }' if not search else '{ "alumnus": "No Matches!" }')

    return concat_file.getvalue()

Answer 1

I suspect this line in your code:

if not j == alumni.last_names.count() - 1:

(and a few similar lines).

You didn't post your model but to me that looks like alumni.last_names might be a query? Running a query for each entity would be a super bad idea, and might very well dominate your cost. It should not take anywhere near 30 seconds to concatenate a few thousand strings using cStringIO.

It's easy to find out if you are doing too many queries using Appstats: http://code.google.com/appengine/docs/python/tools/appstats.html (you can even try this in the dev appserver).

PS. The singular is actually alumnus and the plural is alumni. :-)

Answer 2

str.join() and string interpolation usually give much better performance than repeated concatenation. Give those a try, and may the powers that be have mercy on your soul.

Answer 3

I'd suggest to create the data structure that you want to send as an answer in python itself and use a json module to generate the string version. The reason for this is that the most popular json modules are at least partly implemented in c so, despite cStringIO is also implemented in c, I guess they do some optimizations difficult to achieve using just the standard library.

For more information, please refer to this related question.

Edit: If using an third party json module is out of the question, then I'd try to reduce the number of write calls by using a formatting string as long as possible.

I guess using a templating library that could speed that up would be also ruled out, so the only way that I can think of would be to cache as much as possible, so that subsequent calls don't need to redo the whole task.