making a batches using for loop in python

Question

I did the following code, but did not get the expected output.

a=[1, 2, 3, 4, 5, 5, 6, 7, 78, 8, 9, 9, 0, 0, 8, 6, 5, 4, 3, 4]


def chunks(lst, n):
    """Yield successive n-sized chunks from lst."""
    for i in range(0, len(lst), n):
        yield lst[i:i + n]
        

def hello():
    s=chunks(a,10)
    print(list(s))

hello()  

#output [[1, 2, 3, 4, 5, 5, 6, 7, 78, 8], [9, 9, 0, 0, 8, 6, 5, 4, 3, 4]]

expected output-:
[1, 2, 3, 4, 5, 5, 6, 7, 78, 8]
---------New line----
[9, 9, 0, 0, 8, 6, 5, 4, 3, 4]

expected out is only possible if I am able to put a return in for loop, which is not possible. I think.

Thanks in advance.

that output return "as a whole", I want batches(chunks). and I have to do this otherwise I get this ` @DaniMesejo ` — Coder, Oct 25 '21 at 21:18
@j1-lee Thanks for the comment, could you help me with that. i gave it try, but not getting what I am looking for. if it is easy for you, kindly help me out! — Coder, Oct 25 '21 at 21:21
@Coder I guess Dani Mesejo's comment is a right answer; how about trying that? Good luck! :) — j1-lee, Oct 25 '21 at 21:22
@DaniMesejo, it is splitting the list after I get whole the output. I want to return a single batch after every iteration. — Coder, Oct 25 '21 at 21:26
@j1-lee no, no! it is a different thing than what I am looking for. — Coder, Oct 25 '21 at 21:26
since s is generator object, when you list it, it iterates over all the elements and making a list. that is why you see your chunks in one list. Instead, you should loop over your chunks as @DaniMesejo indicated. — aykcandem, Oct 25 '21 at 21:27
What is the difference of the output vs the expected output. Please try to be explicit and clear, we don't have as much context about the problem as you do. — Dani Mesejo, Oct 25 '21 at 21:30
@DaniMesejo let me explain a bit more. right now, Output is a list of lists, that can be converted into an expected output as you suggested and people suggest as well (this is what I don't want). both look the same. But, I want to return a list directly from the Fun(chunk). I don't want to split after I get the output. let's say n=5, which means I want to return 4 lists(one after another) in each iteration. --- `s` is a list of lists, then we can separate it later, that I dont want to do. is it clear? — Coder, Oct 25 '21 at 21:43
I think it is still not clear. Can you try to simplify it a little ? Just write down how you want to get the return. The exact output you want. — Prashant Kumar, Oct 25 '21 at 21:48
@PrashantKumar the expected output is the same. it can be achieved using what you and people already suggested. I do not want to apply the `loop` or `next`. i want to return those lists (which we have in `s`) directly from the `chunk`. E.g N=10, in the first call I want to return `[1, 2, 3, 4, 5, 5, 6, 7, 78, 8]`, in second call `[9, 9, 0, 0, 8, 6, 5, 4, 3, 4]`. i don't want to separate it in the end, that thing I already know. is it clear? — Coder, Oct 25 '21 at 21:58
@Coder that is exactly what you achieve with next(). s is generator object, not a list. So, your chunks are not defined yet. You just have the generator object. When you first call next(), your first chunk is created and returned. Second chunk still not created. However, generator object knows that you already call it once. So, the next chunk will be second 10 numbers in your list. When you do second next() call, next chunk will be returned. In a nutshell, you do not separate a list. You just create each chunk, whenever you call next(). This looks like same with your expectation, isnt it? — aykcandem, Oct 25 '21 at 22:06

score 1 · Answer 1 · answered Oct 25 '21 at 21:31

Use next() method for getting elements of generator one by one.

a=[1, 2, 3, 4, 5, 5, 6, 7, 78, 8, 9, 9, 0, 0, 8, 6, 5, 4, 3, 4]


def chunks(lst, n):
    """Yield successive n-sized chunks from lst."""
    for i in range(0, len(lst), n):
        yield lst[i:i + n]
        

def hello():
    s=chunks(a,10)
    # s is a generator object
    # to print the chunk use next method
    print(next(s))
    print(next(s))

hello()

Output

[1, 2, 3, 4, 5, 5, 6, 7, 78, 8]
[9, 9, 0, 0, 8, 6, 5, 4, 3, 4]

i added comment in the above section, kindly take a look! – Coder Oct 25 '21 at 21:44 — Coder, Oct 25 '21 at 21:44

Dani Mesejo · Accepted Answer · 2021-10-26T08:39:18.343

If I understood correctly you want a function chunks that does the following:

a = [1, 2, 3, 4, 5, 5, 6, 7, 78, 8, 9, 9, 0, 0, 8, 6, 5, 4, 3, 4]
print(chunks(a, 10))  # [1, 2, 3, 4, 5, 5, 6, 7, 78, 8]
print(chunks(a, 10))  # [9, 9, 0, 0, 8, 6, 5, 4, 3, 4]

The rest of the answer is based on that assumption.

TL;DR

It cannot (should not) be done with simple functions, use a generator function + next

Long Explanation

To the best of my knowledge you cannot use a simple function for that, basically because you cannot keep state in a function.

You could use the following tricks (at your own peril :)):

from itertools import count

a = [1, 2, 3, 4, 5, 5, 6, 7, 78, 8, 9, 9, 0, 0, 8, 6, 5, 4, 3, 4]


def chunks(lst, n, counter=count()):
    """Yield successive n-sized chunks from lst."""
    i = next(counter) * n
    return lst[i:i + n]


print(chunks(a, 10))
print(chunks(a, 10))

Output

[1, 2, 3, 4, 5, 5, 6, 7, 78, 8]
[9, 9, 0, 0, 8, 6, 5, 4, 3, 4]

Note that is generally frowned upon because the argument counter the function chunks is a default mutable argument. Another alternative is to use a closure:

from itertools import count

a = [1, 2, 3, 4, 5, 5, 6, 7, 78, 8, 9, 9, 0, 0, 8, 6, 5, 4, 3, 4]

def chunks_maker():
    counter = count()

    def _chunks(lst, n):
        """Yield successive n-sized chunks from lst."""
        i = next(counter) * n
        return lst[i:i + n]

    return _chunks


chunks = chunks_maker()

batch = 2
for i in range(batch):
    print(chunks(a, 10))

Output (from closure)

[1, 2, 3, 4, 5, 5, 6, 7, 78, 8]
[9, 9, 0, 0, 8, 6, 5, 4, 3, 4]

But, the Zen of Python states:

There should be one-- and preferably only one --obvious way to do it.

and that way at least in my humble opinion is to use a generator function + next.

Other resources on keeping state without classes in Python are:

score 0 · Answer 3 · answered Oct 25 '21 at 21:23

0

Variable s in your hello method is a generator object. You can simply iterate over the generator and keep on printing the values in new line.

Try this:

a=[1, 2, 3, 4, 5, 5, 6, 7, 78, 8, 9, 9, 0, 0, 8, 6, 5, 4, 3, 4]


def chunks(lst, n):
    """Yield successive n-sized chunks from lst."""
    for i in range(0, len(lst), n):
        yield lst[i:i + n]
        

def hello():
    s=chunks(a,10)
    for l in s:
        print(l)

hello()

Output:

[1, 2, 3, 4, 5, 5, 6, 7, 78, 8]
[9, 9, 0, 0, 8, 6, 5, 4, 3, 4]

answered Oct 25 '21 at 21:23

Prashant Kumar

2,057
2
9
22

thanks for the comment. but this is not what I am looking for. I want to return a single batch after every iteration. directly from the `Fun-chunks`. – Coder Oct 25 '21 at 21:27
Can you explain what do you mean by single batch after every iteration ? – Prashant Kumar Oct 25 '21 at 21:28
the output of the code is returned "as a whole"(list of lists). my use case is, if i pass n=10(batch_size or chunk size), i want it to return the list separately in each iteration. Not the list of lists. – Coder Oct 25 '21 at 21:31
If you print the variable "s" in ```hello``` method, you will see it as a generator object. ```s``` is not a list of list. You can iterate over the generator object using ```next``` keyword as well. – Prashant Kumar Oct 25 '21 at 21:33
i added more details in above comment section, kindly take a look! – Coder Oct 25 '21 at 21:43

making a batches using for loop in python

3 Answers3