0
a = [(1,2),(3,1),(4,4),(5,5),(5,5),(5,5),(5,5),(5,5),(5,5),(5,5),(5,5),(5,5),(5,5),(5,5),(5,5),(5,5),(5,5),(5,5)]
# Quite a lot tuples in the list, 6 digits~
# I want to split it into rows and columns.
rows = 5
cols = 5

Data structure is
rows and cols are the index for the bit list
[rows, cols, (data)]

I use loop to do this, but it takes too long for processing a big amount of tuples.

processed_data = []
index = 0
for h in range(0, rows - 1):
    for w in range(0, cols - 1):
        li = []
        li = [h, w, a[index]]
        processed_data.append(li)
        index += 1

This operation takes too long, is there a way to do optimization? Thanks very much!

wrongusername
  • 18,564
  • 40
  • 130
  • 214
user469652
  • 48,855
  • 59
  • 128
  • 165
  • Any thoughts on approach? What have you tried so far? Why are you optimizing this in particular - are you sure it is the bottleneck in your program (i.e., what percent of the execution time is it taking, and what is your desired target) – Merlyn Morgan-Graham Oct 23 '10 at 02:27
  • @Merlyn Morgan-Graham, it needs to be optimized even if it's not a bottleneck. Creating an empty list just to immediately replace the binding with another brand new list? Not good. – aaronasterling Oct 23 '10 at 02:30
  • @aaronasterling: What you just described sounds like code stink, not an optimization. They are entirely different things, and require different approaches. Code stink you must learn to identify, and avoid creating. Optimizations should only be done once balancing bottlenecks that you can't ship with (that you identified w/ proper tools and realistic scenarios) vs ease of reading the code/ease of future maintenance. – Merlyn Morgan-Graham Oct 23 '10 at 02:35
  • 1
    Why a 2-d data structure, and not a dictionary mapping a tuple to a total count such as: `{(1,2) : 123, (4,5) : 534 ...}`? What about the 6 digits? What are you really trying to accomplish here? What is this for, mate? – Hamish Grubijan Oct 23 '10 at 02:42
  • @aaronasterling: Although you're right, sometimes you will go "why did I do it that way!" and fix it w/o bothering to profile :) – Merlyn Morgan-Graham Oct 23 '10 at 03:19
  • If you want 5 `rows` and `cols`, I think your two for loops would need to be `for h in range(0, rows):` and `for w in range(0, cols):`. Correct? – martineau Oct 23 '10 at 13:05

3 Answers3

2

It's not at all clear to me what you want but here's a shot at the same loop in a more optimized manner:

import itertools as it

index = it.count(0) 
processed_data = [[h, w, a[next(index)]] 
                 for h in xrange(0, rows - 1)
                 for w in xrange(0, cols - 1)]

or, since you've already imported itertools, 

index = ite.count(0)
indices = it.product(xrange(0, rows-1), xrange(0, cols-1))
processed_data = [[h, w, a[next(index)]] for h, w in indices]

The reason that these are faster is that they use list comprehensions instead of for loops. List comprehensions have their own opcode, LIST_APPEND, which routes directly to the append method on the list that's being constructed. In a normal for loop, the virtual machine has to go through the whole processes of looking up the append method on the list object which is fairly pricey.

Also, itertools is implemented in C so if it's not faster for the same algorithm, then there's a bug in itertools.

aaronasterling
  • 68,820
  • 20
  • 127
  • 125
2

Fine, if you really want the indices that badly...

[divmod(i, cols) + (x,) for i, x in itertools.izip(itertools.count(), a)]
Ignacio Vazquez-Abrams
  • 776,304
  • 153
  • 1,341
  • 1,358
0

Sounds like you want to split it into evenly-sized chunks.

Community
  • 1
  • 1
Ignacio Vazquez-Abrams
  • 776,304
  • 153
  • 1,341
  • 1,358
  • no, that shouldn't be right because only one element of the old list ends up in each element of the new list in the code given. – aaronasterling Oct 23 '10 at 02:31
  • @aaronsterling: The only difference is that it doesn't include `h` and `w`. But this isn't necessarily a problem since you can derive them from where you are in the nested structure via `enumerate()`. – Ignacio Vazquez-Abrams Oct 23 '10 at 02:34
  • no, the difference is that slices of the list are being returned by the generator you linked too because OP of that question wanted chunks of data. OP of this question wants essentially the same list decorated with additional indices. – aaronasterling Oct 23 '10 at 02:36
  • @aaronsterling: The adornments can be derived on the other side. Yes, it means a little more work on the other side, but it means less work on this side. – Ignacio Vazquez-Abrams Oct 23 '10 at 02:43