How to process list in batches of N elements, without missing the last ones?

Question

Sorry for wording of the question.

I am reading a file and after some operation I am adding that line to a list. I want to read 100 lines, append it to list then send it and repeat the process.

I've been doing line_count % 100 == 0 then send my list however final 34 lines of code will not get sent, how can I achieve this?

Minimal code to reproduce:

lst = [1,2,3,4,5,6,7,8,9,10,11]

line_count = 0

final_lst = []
for line in lst:
    line_count += 1
    final_lst.append(line)
    if line_count % 2 == 0:
        print(f"succesfully sent:{final_lst}")

output:

succesfully sent:[1, 2]
succesfully sent:[1, 2, 3, 4]
succesfully sent:[1, 2, 3, 4, 5, 6]
succesfully sent:[1, 2, 3, 4, 5, 6, 7, 8]
succesfully sent:[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

if you print, it will succesfully send up to [1,2,3,....,10]

Please read [ask]. I have _no idea_ what you're trying to ask. — ChrisGPT was on strike, Dec 05 '19 at 01:14
reason I didn't post my code is because it contains a lot of info that is irrelevant, I've added a minimal reproducible code. Thanks for the feedback. — haneulkim, Dec 05 '19 at 01:17
@lmbloo Here nobody has asked you to provide the code of your project, I have only been asked to provide an MRE that are different things but in the case that your project is small both can be the same. On the other hand, since the MRE does not reflect what you indicate in the description of your problem, then you must explain that part of the output is incorrect and which is the desired output. — eyllanesc, Dec 05 '19 at 01:19
Can you provide a bit more context for this? Depending on what your program is doing there should be a few different solution. — AMC, Dec 05 '19 at 02:46
Does this answer your question? [How do you split a list into evenly sized chunks?](https://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-evenly-sized-chunks) — AMC, Dec 05 '19 at 02:48

ruohola · Accepted Answer · 2019-12-05T01:26:03.330

2

You could you simply add a check to do the "sending" always also on the final iteration of the loop:

lst = [1,2,3,4,5,6,7,8,9,10,11]

line_count = 0

final_lst = []
for line in lst:
    line_count += 1
    final_lst.append(line)
    if line_count % 2 == 0 or line_count == len(lst):
        print(f"succesfully sent:{final_lst}")

Output:

succesfully sent:[1, 2]
succesfully sent:[1, 2, 3, 4]
succesfully sent:[1, 2, 3, 4, 5, 6]
succesfully sent:[1, 2, 3, 4, 5, 6, 7, 8]
succesfully sent:[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
succesfully sent:[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]

We can also simplify the code a bit if we use enumerate:

lst = [1,2,3,4,5,6,7,8,9,10,11]

final_lst = []
for line_count, line in enumerate(lst, start=1):
    final_lst.append(line)
    if line_count % 2 == 0 or line_count == len(lst):
        print(f"succesfully sent:{final_lst}")

edited Dec 05 '19 at 01:26

answered Dec 05 '19 at 01:19

ruohola

21,987
6
62
97

This is exactly what @sammy said lol, but thanks – haneulkim Dec 05 '19 at 01:20
This will print the same list twice if the number of elements is a multiple of 2. Not sure if this a problem for the OP though. – Selcuk Dec 05 '19 at 01:21
@Selcuk Nice point, added a safeguard for that! – ruohola Dec 05 '19 at 01:23
@ruohola You could've changed it to `or line_count == len(lst)` and gotten rid of the final `print`. – Selcuk Dec 05 '19 at 01:25
@Selcuk Nice point again :D – ruohola Dec 05 '19 at 01:27
adding another condition `line_count == len(lst)` seems much more cleaner. Thanks! – haneulkim Dec 05 '19 at 01:33
2

Glad I could provide an answer where @Selcuk could help :) – ruohola Dec 05 '19 at 01:33

score 1 · Answer 2 · answered Dec 05 '19 at 01:16

1

You should probably perform an additional final "sending operation" after leaving the code block of the for loop.

answered Dec 05 '19 at 01:16

sammy

857
5
13

How to process list in batches of N elements, without missing the last ones?

2 Answers2