Python splitting big lists into smaller lists for check performance?

Question

What is the best way to split a large list (over 1000 items) for performance purposes? And use "parts" of a large list?

For example; I have a list of 10k addresses I want to scan. How can I split this list into a 4th, and finish the list by sections to increase performance?

Is [this](http://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-evenly-sized-chunks-in-python) question similar to what you want? — thegrinner, Jul 06 '11 at 16:13
What are you doing with the list? Why do you expect splitting the list to help with performance? — Karl Knechtel, Jul 06 '11 at 16:25

phihag · Answer 1 · 2011-07-06T17:02:58.643

2

If you're reading all elements of the list, splitting will actually decrease your performance. You should profile your program (maybe post it in a question here, if you can sufficiently simplify it).

If you're not reading all elements, why are you using a list in the first place? Use a set or dict. Operations on both are extremely fast.

edited Jul 06 '11 at 17:02

answered Jul 06 '11 at 16:19

phihag

278,196
72
453
469

score 1 · Answer 2 · edited May 23 '17 at 12:19

1

See the answers for this question for how to split an arbitrary list into smaller chunks. It doesn't explicitly cover increasing performance, so you might want to consider looking into the multiprocessing module for actually manipulating those small lists simultaneously if that's what you're looking for.

edited May 23 '17 at 12:19

Community

1
1

answered Jul 06 '11 at 16:19

thegrinner

11,546
5
41
64

score 0 · Answer 3 · edited May 23 '17 at 11:55

0

So you want to do an operation on the whole list, but it's slow doing it sequentially? I think you want a worker pool. Does this help?

edited May 23 '17 at 11:55

Community

1
1

answered Jul 06 '11 at 16:17

Daenyth

35,856
13
85
124

Python splitting big lists into smaller lists for check performance?

3 Answers3