Why is unpack so slow?

Question

I have a simple example:

def __init__(self,string):
    self.string = string

def UI32(self):
    tmp = self.string[:4]
    self.string = self.string[4:]
    return unpack(">I",tmp)[0]

data = file.read()
U = UI(data)
for i in range(60000):
    test = UI32()

Total time: 22 seconds!

Answer 1

First of all, I cannot reproduce the 22s on my system (Intel Nehalem, 64-bit Ubuntu, Python 2.6.5).

The following takes 1.4s (this is essentially your code with some blanks filled in by me):

import struct

class UI(object):
    def __init__(self,string):
        self.string = string

    def UI32(self):
        tmp = self.string[:4]
        self.string = self.string[4:]
        return struct.unpack(">I",tmp)[0]

U = UI('0' * 240000)
for i in range(60000):
    test = U.UI32()

Now, there are several glaring inefficiencies here, especially around self.string.

I've rewritten your code like so:

import struct

class UI(object):
    def __init__(self,string):
        fmt = '>%dI' % (len(string) / 4)
        self.ints = struct.unpack(fmt, string)
    def __iter__(self):
        return iter(self.ints)

U = UI('0' * 240000)
count = 0
for test in U:
    count += 1
print count

On the same machine it now takes 0.025s.

Answer 2

Every iteration of the 60,000 cycle loop you are copying the entire memory buffer:

self.string = self.string[4:]

It would be more efficient to simply walk through the string using indexes and at the end clear the variable.