How do I make string-like variable that iterates over a large string?

Question

In Python, can I build a variable that acts like a string but is internally iterating through a sequence of strings?

For instance

def function_a():
    for i in xrange(100000000):
        yield str(i)

This, will iterate over a list of strings and it will do it efficiently - keeping only one string in memory at a time. But what I want is something like this:

''.join([s for s in function_a()])

But I bet this just does the naïve thing and iterates through the entire set and concatenates them all into one big string in memory. The other problem with this, is that I want a variable, I don't want to have to expose the user to the ugly work of actually doing the join. So maybe the user would do something like:

magic_str = get_long_but_memory_efficient_str()

And then use it to efficiently print to the screen (and free up memory as it goes):

print magic_str

Or my real use for it is to HTTP stream to a server:

request = urllib2.Request(url, magic_str)

Apparently something like this exists. Check out the code below for efficiently streaming a file to a server (from this question).

f = open('somelargefile.zip','rb')
mmapped_file_as_string = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
request = urllib2.Request(url, mmapped_file_as_string)
request.add_header("Content-Type", "application/zip")
response = urllib2.urlopen(request)

But my case is different because I'm constructing the string that I'm streaming to the server.

Anuj Gupta · Answer 1 · 2013-03-01T20:04:56.127

Updated answer for your practical need:

>>> class MagicString(str):
    def __init__(self, gen):
        self.gen = gen
    def __str__(self):
        try:
            return self.gen.next()
        except StopIteration:
            return '' #boolean value = False


>>> def run_efficiently(some_function, magic_str):
    substr = str(magic_str)
    while substr:
        some_function(substr)
        substr = str(magic_str)

Explanation: You need a combination of:

An object that yields substrings using a generator (e.g. on string representation)
A function that calls another function until the object's representation returns an empty string.

Extending this example to print:

>>> import sys
>>> def print_without_breaks(some_string):
    sys.stdout.write(some_string)

>>> s = MagicString(c for c in '12345')
>>> run_efficiently(print_without_breaks, s)
12345

You can use a similar one for your practical need where you can do something useful with the returned values of each request.

So maybe, you don't need a variable/object at all... Just some simple code that runs repeatedly until your generator spits out a StopIteration exception.

That's really close! Just replace the contents of __str__ with `return ''.join(self.gen)` and it's functionally just what I want. The next question is "Is this efficient?" So with my modification, does `print a` create the entire string in memory and then print it? Or does print iterate through the contents of the string and print it one char at a time (while in the background the string is being generated). — JnBrymn, Mar 01 '13 at 18:35
If you do a join, it will store the complete the joined string in memory before printing it. If you want to print a string by printing its substrings one at a time, you'll want to redefine `print` to call the generator (or in this case, use `a`) repeatedly. — Anuj Gupta, Mar 01 '13 at 18:42

score 0 · Answer 2 · answered Mar 01 '13 at 18:12

Not sure I understood exactly what you want, but it seems to me that you are concerned over the immutability of python strings.

join won't create a lot of temporary objects as you think. If you already have a list, ''.join is going to be pretty efficient and will create just one single string.

If you have no reason to create a list with the object you want to concat, just use cStringIO module. This is going to use the lest memory.

If you are still concerned or either you are die hard C programmer that don't understand how people cannot see that null terminated sequences of bytes are the way God wanted us to deal with strings, write that portion of your code in C, this is something that is pretty use to do in python, compared for example, with Java.

Actually, in `Cpython` (at least), `''.join(x for x in whatever)` is (more or less) equivalent to `''.join(tuple(whatever))`. So, OP is right about that part. — mgilson, Mar 01 '13 at 18:14
If ''.join creates "one single string", it's not efficient for my purposes. Sure, it's fast, but I don't want to hold the entire string in memory at once. Memory consumption is my main concern. — JnBrymn, Mar 01 '13 at 18:38

How do I make string-like variable that iterates over a large string?

2 Answers2