Python go to specific iteration in generator

Question

I'm looking for a way to navigate to a particular iteration in a generator object.

I have a generator object that goes through a list of JSON objects. Instead of loading all of them at once, I created a generator so that each JSON object loads only at each iteration.

def read_data(file_name):
    with open(file_name) as data_file:
        for user in data_file:
            yield json.loads(user)

However, now I am looking for some way to navigate to the nth iteration to retrieve more data on that user. The only way I can think of doing this is iterating through the generator and stopping on the nth enumeration:

n = 3
data = read_data(file_name)
for num, user in enumerate(data):
    if num == n:
        <retrieve more data>

Any better ways to do this?

That's fundamentally how you have to do it, but you should probably use `itertools.islice()`. Can't do a full answer; I'm on my phone :P. — Cyphase, Aug 21 '15 at 01:42
You could / should store the results in a `list` if you're going back and forth in the results. — Cyphase, Aug 21 '15 at 01:47
@Cyphase is there a way to create a dictionary? maybe of objects that point to the iteration? — Aaron Yan, Aug 21 '15 at 02:23
You could create an object with a 'dict'-like interface, but you can't reverse an iterator; you'd have to store the past values. There might be a better way to do what you want though. Just let me get to my computer :). — Cyphase, Aug 21 '15 at 02:39
Why are you trying to do this? How much data are you working with? — Cyphase, Aug 21 '15 at 03:04
@Cyphase oh dang, ya storing is not an option. I'm currently not using a lot of data but my work should be able to scale to larger data sets (millions of users) — Aaron Yan, Aug 21 '15 at 03:38
I'm only asking to make sure that you do actually need more sophisticated methods :). — Cyphase, Aug 21 '15 at 03:42
I'm gonna start working on something to do what I think you want; sounds like fun :). — Cyphase, Aug 21 '15 at 03:47
Just to mention though, depending on your exact usage, you might be able to use a database to do this pretty easily. — Cyphase, Aug 21 '15 at 03:51
@Cyphase Aww, a database might be what I'm missing here! map each user with a key and then call the database key!! — Aaron Yan, Aug 21 '15 at 16:44

Cyphase · Accepted Answer · 2015-08-21T02:08:19.267

2

This should do it:

from itertools import islice

def nth(iterable, n, default=None):
    "Returns the nth item or a default value"
    return next(islice(iterable, n, None), default)

This is one of many useful utilities included in the itertools documentation.

edited Aug 21 '15 at 02:08

answered Aug 21 '15 at 01:51

Cyphase

11,502
2
31
32

Thanks for fixing the indentation Anand; I'm on my phone and I accidentally posted without properly formatting it :P. – Cyphase Aug 21 '15 at 01:55
2

Thanks Padraic. Using my phone is no excuse for sloppy answers :). – Cyphase Aug 21 '15 at 02:09
works nicely. i'll consider mapping each user to a database key if performance doesn't meet my requirements. thanks for the help! – Aaron Yan Aug 21 '15 at 16:48

Python go to specific iteration in generator

1 Answers1