Python - Why does list(set(my_list)) destroy the original order of the data?

Question

I am trying to remove the duplicates in a list my_list

In [13]: my_list
Out[13]: [[(-7.7725138974675732, -79.168560840024711),
  (-7.7725138974675732, -79.168560840024711),
  (-5.5, -59.0),
  (-4.4, -48.3),
  (-3.2, -37.5),
  (-2.1, -26.9),
  (-0.9, -16.2),
  (2.7974572775821454, 4.4251549647685868),
  (2.7974572775821454, 4.4251549647685868)],
 [(2.7974572775821454, 4.4251549647685868),
  (2.7974572775821454, 4.4251549647685868),
  (29.8, 16.4),
  (41.8, 16.9),
  (53.1, 17.2),
  (74.767340748230566, 15.947522750252775),
  (74.767340748230566, 15.947522750252775),
  (74.767340748230566, 15.947522750252775),
  (97.8, 13.4),
  (109.2, 14.9),
  (125.57323627494615, 19.53393479479609)],
 [(68.773124224303274, -79.616585806160444),
  (68.773124224303274, -79.616585806160444),
  (71.141449110088374, -58.675607868692225),
  (73.874255065554237, -27.94398092967819),
  (73.874255065554237, -27.94398092967819),
  (73.874255065554237, -27.94398092967819),
  (74.5, -2.9),
  (74.767340748230566, 15.947522750252775)],
 [(-109.09700131981562, -32.09602243732722),
  (-108.3, -32.7),
  (-81.9, -52.7),
  (-64.94343179027949, -62.943395681375023),
  (-64.94343179027949, -62.943395681375023),
  (-44.8, -70.5),
  (-32.2, -74.1),
  (-7.7725138974675732, -79.168560840024711),
  (-7.7725138974675732, -79.168560840024711),
  (10.6, -80.1),
  (21.6, -80.4),
  (32.8, -80.6),
  (43.9, -80.6),
  (68.773124224303274, -79.616585806160444),
  (68.773124224303274, -79.616585806160444),
  (68.773124224303274, -79.616585806160444),
  (94.59403067734813, -75.99089907199837)],
 [(-57.743037814594651, 0.48281269760971313),
  (-57.743037814594651, 0.48281269760971313),
  (-61.5, -30.9),
  (-62.9, -42.2),
  (-64.94343179027949, -62.943395681375023),
  (-64.94343179027949, -62.943395681375023)],
 [(73.874255065554237, -27.94398092967819),
  (73.874255065554237, -27.94398092967819),
  (113.3911596081064, -33.232674908171724)],
 [(71.141449110088374, -58.675607868692225),
  (71.141449110088374, -58.675607868692225)],
 [(2.7974572775821454, 4.4251549647685868),
  (-17.6, 3.5),
  (-57.743037814594651, 0.48281269760971313),
  (-57.743037814594651, 0.48281269760971313)]]

Then, I do my_list = list(set(my_list))

In [20]: my_list
Out[20]: [[(-7.7725138974675732, -79.168560840024711),
  (-3.2, -37.5),
  (-2.1, -26.9),
  (-5.5, -59.0),
  (-4.4, -48.3),
  (-0.9, -16.2),
  (2.7974572775821454, 4.4251549647685868)],
 [(109.2, 14.9),
  (41.8, 16.9),
  (97.8, 13.4),
  (29.8, 16.4),
  (53.1, 17.2),
  (125.57323627494615, 19.53393479479609),
  (74.767340748230566, 15.947522750252775),
  (2.7974572775821454, 4.4251549647685868)],
 [(73.874255065554237, -27.94398092967819),
  (74.5, -2.9),
  (74.767340748230566, 15.947522750252775),
  (68.773124224303274, -79.616585806160444),
  (71.141449110088374, -58.675607868692225)],
 [(-108.3, -32.7),
  (-7.7725138974675732, -79.168560840024711),
  (-44.8, -70.5),
  (-81.9, -52.7),
  (10.6, -80.1),
  (-64.94343179027949, -62.943395681375023),
  (94.59403067734813, -75.99089907199837),
  (-109.09700131981562, -32.09602243732722),
  (68.773124224303274, -79.616585806160444),
  (32.8, -80.6),
  (-32.2, -74.1),
  (21.6, -80.4),
  (43.9, -80.6)],
 [(-62.9, -42.2),
  (-64.94343179027949, -62.943395681375023),
  (-57.743037814594651, 0.48281269760971313),
  (-61.5, -30.9)],
 [(73.874255065554237, -27.94398092967819),
  (113.3911596081064, -33.232674908171724)],
 [(71.141449110088374, -58.675607868692225)],
 [(-17.6, 3.5),
  (-57.743037814594651, 0.48281269760971313),
  (2.7974572775821454, 4.4251549647685868)]]

As can be seen, the order of the data has changed. The order is important for me. So I have to maintain the order after the operation.

Why is it so and how do I maintain the order?

score 3 · Answer 1 · edited May 23 '17 at 12:31

3

the order is destroyed because of set, which by nature is an un-ordered collection (edit, thx)

as to maintaining the order: How do you remove duplicates from a list in whilst preserving order?

edited May 23 '17 at 12:31

Community

1
1

answered Oct 19 '13 at 02:05

Cameron Sparr

3,925
2
22
31

Then how may I maintain the order? – Oct 19 '13 at 02:06

score 3 · Accepted Answer · edited May 23 '17 at 11:52

3

As @Cameronsparr says, you ordering because sets are inherently unordered.

To maintain the order I would do something like this:

old_list = [...]
new_list = []
tmp_set = set()
for el in old_list:
     if el not in tmp_set:
          tmp_set.add(el)
          new_list.append(el)

How do you remove duplicates from a list in whilst preserving order? has a much-slicker way of doing this.

edited May 23 '17 at 11:52

Community

1
1

answered Oct 19 '13 at 02:10

tacaswell

84,579
22
210
199

Thanks but why both solutions give `TypeError: unhashable type: 'list'`? – Oct 19 '13 at 02:18
@mavErick because lists are not hashable, which is a requirement for addings things to a set. Convert your lists to tuples. – tacaswell Oct 19 '13 at 02:37
I've converted the old list to tuple, but it somehow still reports this error. http://pastie.org/8413483 – Oct 19 '13 at 02:44
1

if this didn't solve your problem you should not have accepted the answer. – tacaswell Oct 19 '13 at 02:52
But the problem that remains is only a syntax one. The concept is correct. So I accept it. – Oct 19 '13 at 02:54
You should ask a new question with the code in that link. It it too long to explain what is wrong in the comments section. You are only changing in locally in one place. – tacaswell Oct 19 '13 at 02:57
let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/39539/discussion-between-tcaswell-and-maverick) – tacaswell Oct 19 '13 at 04:14

Python - Why does list(set(my_list)) destroy the original order of the data?

2 Answers2