0

I'm trying to use Python/NumPy to sort a "list of lists" which consist of sublists of "row, col, value" extracted for a series of numpy array elements that I've zipped out to a list. I'm trying to sort on the 3rd item in these sublists and an itemgetter sort works fine on relatively small lists, but as i get in to lists of millions of elements, the sort fails.

I'm using:

zip(*np.where(detected_locations))

To create an initial list of row/col values, which I'm iterating through to extract the array element and append to each sublist, leading to an unsorted list of lists...

[(0, 108, 94.16000366210938), (0, 131, 94.8499984741211), (0, 177, 95.41000366210938), (0, 210, 95.63999938964844), (0, 250, 95.75), (0, 256, 95.87999725341797), (0, 269, 95.97000122070312), ...

I'd like to sort on the 3rd element, to this list:

[(375, 722, 80.83000183105469), (375, 727, 80.87999725341797), (378, 728, 80.87999725341797), (373, 717, 80.9000015258789), (374, 725, 80.9000015258789), (378, 723, 80.9000015258789),

This is the code that works fine on smaller lists, but doesn't work on larger lists:

 SortedList = sorted(theList,key=itemgetter(2))

Can anyone suggest an alternate solution? All I need to do is be able to iterate through each row/col entry sorted ascending order for the 3rd element.

Jae
  • 143
  • 5
  • How does that code fail for larger lists? It sounds like you have some other bug you haven't realized. – user2357112 Sep 22 '16 at 20:14
  • @user2357112 I can only assume it's an out of memory situation because the program shuts down. It's being used in conjunction with ESRI arcpy python site package which should be able to handle the operation in the 64-bit space. Would you expect the code above to function properly with 10+ mill. element lists? – Jae Sep 22 '16 at 20:17
  • It should work - the sizelimit of a list is [somewhere over half a billion entries](https://stackoverflow.com/questions/855191/how-big-can-a-python-array-get). If you have sufficient computational ressources this should work. Did your code fail with the `Killed` message - then it's most likely a memory issue. – Maurice Sep 22 '16 at 20:32
  • You're right. I was overlooking something. In my testing I was trying to print the final sorted list; not iterated by element, but all at once. This is where the failure occurred. Thanks @user2357112 – Jae Sep 23 '16 at 22:56

0 Answers0