Python - Check progress of dictionary creation

Question

I'm creating a dictionary this way:

num_states = 26565000
num_actions = 7

self.P = {
            state: {action: [] for action in range(num_actions)}
            for state in range(num_states)
        }

It's taking quite a bit and I would like to insert print(state) to check the progress of the creation. I'm not sure where to put it.

EDIT:

I need this dictionary beacause I need to involve the whole observation space for an agent I'm trying with reinforcement learning

EDIT.2:

I usually use the regular for loops:

for state in range(num_states)
    print(state)
    for action in range(num_actions)
        ...

I'm not sure how to achieve the same result using this way of coding a for loop.

Do you really need to create a dict entry for every *possible* value of `state`? That's a huge dictionary. Why not create entries if and when you actually need them? — Tom Karzes, Jul 22 '23 at 19:44
This doesn't look right. What are you actually trying to do? — Martynas Žiemys, Jul 22 '23 at 19:45
Does this answer your question? [Progress bar for a "for" loop in Python script](https://stackoverflow.com/questions/43259717/progress-bar-for-a-for-loop-in-python-script) — shaik moeed, Jul 22 '23 at 19:45
@TomKarzes Yes I need all of them, I'm training an agent with q-learning and I need a matrix that involves the whole observations space — Maurizio Brini, Jul 22 '23 at 19:47
Any chance you could get away with making the accesses of `P` be `P[state, action]` rather than `P[state][action]`. You could then just write `self.P = collections.defaultdict(list)` and the entries would be created as you needed them. — Frank Yellin, Jul 22 '23 at 19:49
@shaikmoeed It's the use of this kind of for loop I don't quite understand. I usually use the classic for loops. The one in the question confuses me a bit — Maurizio Brini, Jul 22 '23 at 19:50
There's no place to put a `print()` statement in a dict comprehension like this. You'll have to use a regular `for` loop instead. — John Gordon, Jul 22 '23 at 19:50
@JohnGordon I edited the question, I would I achieve the same result with a regular for loop? — Maurizio Brini, Jul 22 '23 at 19:53
Here's an example of how to use tqdm with a dictionary comprehension: https://pastebin.com/idBWyu4a It's pretty much the same as the way you use it with a for-loop: put the tqdm() around the range(). — Nick ODell, Jul 22 '23 at 20:08
@NickODell That's great, if you want you can post this as an answer so that I can checkmark it — Maurizio Brini, Jul 22 '23 at 20:12
What is the reason to work so hard to keep the hard to read dict comprehension? — Martynas Žiemys, Jul 22 '23 at 20:18
It's a genuine question. I have little experience in programming and trying to learn as well. Is there a reason? Also why use another module for something that seems so simple to achieve on its own? — Martynas Žiemys, Jul 22 '23 at 20:24
@MartynasŽiemys I'm new to programming too and I found the "non-regular" for-loop on a tutorial, I didn't know how to turn it into a regular for-loop. — Maurizio Brini, Jul 22 '23 at 20:31
I think these are kind of important to learn about. They are [list or dictionary comprehensions](https://www.w3schools.com/python/python_lists_comprehension.asp). They might be easier to type in some cases. What you have is a nested one, so it's harder to read. The way I understand readability is kind of a big deal in programing because you spend most of your time reading code, so it should have priority and one should attempt to avoid making things unnecessarily complicated. I think same should go for useless module imports, but I have too little experience not to question my own judgment. — Martynas Žiemys, Jul 22 '23 at 20:41
Actually tdqm seems like a really cool and useful thing in general. Probably not in this situation though. The dictionary takes around 12 GB in RAM with the empty lists it creates - looks like you are going to have fun with it. — Martynas Žiemys, Jul 22 '23 at 20:58
@JohnGordon In Python 3 it's trivial. Are you thinking of Python 2? — Kelly Bundy, Jul 22 '23 at 21:58
@MartynasŽiemys Their dict comp is easy to read. Why not tqdm here? Using that was my first thought. Trivial to use, likely more efficient, doesn't spam the screen. — Kelly Bundy, Jul 22 '23 at 22:02
@KellyBundy No, I wasn't thinking of Python 2. How would one put a print statement inside of a comprehension? — John Gordon, Jul 22 '23 at 22:03
@JohnGordon Not a print *statement*. Can't put a statement in there (except maybe with `exec` or so). But in Python 3 it's not a statement anymore, that's why I asked that. An easy way is `{... for state in range(num_states) if not print(state)}`. — Kelly Bundy, Jul 22 '23 at 22:09
@MartynasŽiemys Likely, yes. Printing is usually *very* slow. Presumably that's why you deviated from their intention of printing every single state? — Kelly Bundy, Jul 22 '23 at 23:34
I timed it out of curiosity. tqdm happens to be slower than printing every 1 in 10000, but that's random, it's definitely faster if every single one is printed. — Martynas Žiemys, Jul 23 '23 at 00:09
@MartynasŽiemys Yeah, both tqdm and print-1-in-10000 should add insignificant overhead here. Whereas print-every might add *massive* overhead and be much slower than the actual dict creation. — Kelly Bundy, Jul 23 '23 at 00:25

Martynas Žiemys · Accepted Answer · 2023-07-23T00:19:55.593

1

num_states = 26565000
num_actions = 7
 
P = {}
for state in range(num_states):
  P[state] = {action: [] for action in range(num_actions)}
  if state % 10000 == 0: #print only once on every 1 in 10000
     print(f"\r{state/num_states:.2%}", end="")

edited Jul 23 '23 at 00:19

answered Jul 22 '23 at 20:16

Martynas Žiemys

316
1
8

or `print(f"\r{state/num_states:.2%}", end="")` to use single line – Martynas Žiemys Jul 23 '23 at 00:20

score 0 · Answer 2 · answered Jul 22 '23 at 22:15

One way to hack it in there, printing only one in 10000 like Martynas' answer:

num_states = 26565000
num_actions = 7

self.P = {
    state: {action: [] for action in range(num_actions)}
    for state in range(num_states)
    if state % 10000 or not print(state)
}

Python - Check progress of dictionary creation

2 Answers2