1

I would like to implement this:

asset_hist = []
for key_host, val_hist_list in am_output.asset_history.items():
     for index, hist_item in enumerate(val_hist_list):
        row = collections.OrderedDict([("computer_name", key_host), ("id", index), ("hist_item", hist_item)])
        asset_hist.append(row)

Using list comprehension. This is as close as I can get:

asset_hist = [collections.OrderedDict([("computer_name", key_host), ("id", index), ("hist_item", hist_item)]) for index, hist_item in enumerate((val_hist_list, key_host)[0]) for key_host, val_hist_list in am_output.asset_history.items()]
Guillaume Jacquenot
  • 11,217
  • 6
  • 43
  • 49
gunslingor
  • 1,358
  • 12
  • 34
  • 6
    Don't? Readability is far more important than brevity. – Sam Craig Jul 21 '17 at 17:14
  • 2
    Will you, or someone, ever have to read it afterwards? If you have trouble writing it.... – Thierry Lathuille Jul 21 '17 at 17:14
  • I completely understand your concern... but read this, what led me to this question since list comprehension doesn't use append: https://stackoverflow.com/questions/45240116/python-orderdict-sputtering-as-compared-to-dict?noredirect=1#comment77447520_45240116 – gunslingor Jul 21 '17 at 17:15
  • You have the for-clauses are reversed in your comprehension. They should be in the same order as your nested-for-loop. – juanpa.arrivillaga Jul 21 '17 at 17:16
  • @user3656612: What part of that question made you think a list comprehension would be in any way helpful? – user2357112 Jul 21 '17 at 17:19
  • @user3656612 what? Why do you care if you use `.append` or not? Essentially, the same exact operation is happening under the hood. Your problem in that other quesiton seems to be you are using an `OrderedDict`, which prior to Python3.6, was nowhere near as compact as a normal dict. In Python 3.6, it is a thin-wrapper around a normal dict, and it is *more* compact on average. Prior to that, an ordered-dict essentially carried around a list with it... – juanpa.arrivillaga Jul 21 '17 at 17:19
  • @juanpa.arrivillaga: You might be surprised to learn that they *haven't* changed OrderedDict to rely on the new dict implementation for ordering. It's still using a separate linked list. – user2357112 Jul 21 '17 at 17:24
  • I've read that list comprehension is fasater than looping thru append, only slightly... but I read that it is faster because it doesn't have to use append. Here is the post, I may not be understanding it: https://stackoverflow.com/questions/22108488/are-list-comprehensions-and-functional-functions-faster-than-for-loops – gunslingor Jul 21 '17 at 17:28
  • Ultimately the real problem here is the other question, this is an attempt to figure out a solution. I seems that around the 3.8 millionth append action, appending small orderedDicts, the append function just stops and acts AS IF it were in an infinite loop. – gunslingor Jul 21 '17 at 17:30
  • @user3656612 yes, but your problem is due to stuttering when you move into swap because `OrderedDict`s are taking up too much memory. That won't be helped by changing to a list comprehension. This won't be helped by a comprehension. – juanpa.arrivillaga Jul 21 '17 at 17:34
  • @user2357112 Oh jeez, *really*? That's annoying. I guess it's OK because I never really use `OrderedDict`s. I assume *eventually* it will go in this direction... – juanpa.arrivillaga Jul 21 '17 at 17:35
  • I understand orderdicts are slower and bigger than dicts... but the error I'm getting indicates they are of an order of magnitude 100,000 larger and that can't be true. I mean, when I use a dict it finishes processing in 10 milliseconds, 4 million rows. When I used ordered dicts it stops around 3.8M on the append function... it just stops... I waited 2 hours for it to continue and it was just stuck on append. If it was purely a mem swap issue, I would expect the loop to get slower, not to just stop for 2 hours. There is something wrong with append in this scenario. – gunslingor Jul 21 '17 at 17:38
  • 1
    `.append` is *definitely not your problem*. The list comprehension uses the exact same underlying operation *anyway*, it just doesn't have the method-resolution overhead, which is minimal, and which would not explain it hanging for hours... – juanpa.arrivillaga Jul 21 '17 at 17:39
  • It is definitely pausing inside append... I think it's append + huge dataset + orderdicts + lack of ram that is causing it... I think append or maybe more fundamentally python isn't handling the mem swap correctly. I think it's overlapping issues and I think, since it always stops on append, that is the only place it could be fixed... it doesn't event exception out, it acts like it's in an infinite loop inside the append function. – gunslingor Jul 21 '17 at 17:42
  • Yes, but again, a list comprehension won't help you. **it calls the same underlying functions**. What version, *exactly* are you using. Check out [this discussion](https://stackoverflow.com/a/2480015/5014455). – juanpa.arrivillaga Jul 21 '17 at 17:46
  • I got the list comprehension working, but you are correct it did not solve the problem. according to the terminal (PYTHON35), according to the first line in my software "#!/usr/bin/env python3"... I'm trying" #!/usr/bin/env python3.5" though I never really understood the inner workings of this shabang stuff. – gunslingor Jul 21 '17 at 17:51
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/149837/discussion-between-juanpa-arrivillaga-and-user3656612). – juanpa.arrivillaga Jul 21 '17 at 17:52
  • Chat is blocked at work... heading home to continue discussion... 15 minutes... thank you sir, will be 25 minutes till I get VPN'd back in – gunslingor Jul 21 '17 at 17:54
  • @user3656612 so, you should just cut the gordian knot here and dispense with `dict`s entirely. You are using them as a record, but that isn't really a use-case for dicts. Just an empty dict has a memory overhead of almost 300 bytes. Check out [this answer](https://stackoverflow.com/questions/43578843/many-dictionaries-using-massive-amounts-of-ram/43578969#43578969) to see what I mean. – juanpa.arrivillaga Jul 21 '17 at 18:55

1 Answers1

0

You are close:

import collections

asset_hist = [collections.OrderedDict([("computer_name", key_host), ("id", index), ("hist_item", hist_item)]) for index, hist_item in enumerate(val_hist_list) for key_host, val_hist_list in am_output.asset_history.items()]
Ajax1234
  • 69,937
  • 8
  • 61
  • 102