1

I post this question since all existing answers solve this issue in one dimension using islice, which is not the same (at least I couldn't figure out how to transfer current solutions to three dimensions).

Suppose we have the following script that produces all discrete combinations of integers for three dimensions ranging from (1 to 127):

for j, k, l in itertools.combinations(range(1, 128), 3):
    result = calculations([128, i, j, k, 0])
    writeFile(result+"\n")

and suppose that the script gets interrupted at a random (recorded) point, e.g. [128, 121, 98, 45]. How can we use islice (or any other library) to continue with next iteration, i.e. [128, 121, 98, 46] and onwards until [128,127,126,125]?

Thank you for your time,

G. Stergiopoulos
  • 15
  • 7
  • 1
    first half of [this](https://stackoverflow.com/questions/36802314/python-itertools-product-start-from-certain/36802670#36802670) would probably work, track the indices while you go and specify the starting point on the next run. – Tadhg McDonald-Jensen Feb 11 '22 at 19:21
  • 1
    What do you mean by "interrupted" exactly? – juanpa.arrivillaga Feb 11 '22 at 19:25
  • @TadhgMcDonald-Jensen important to note, that approach would re-iterate over everything that was already seen. – juanpa.arrivillaga Feb 11 '22 at 19:26
  • @juanpa.arrivillaga interrupted means that after a day a power outage + UPS depletion shut down the pc. Nothing fancy, although unrelated to the problem at hand. – G. Stergiopoulos Feb 11 '22 at 23:06
  • Also yes, @TadhgMcDonald-Jensen, reiterating is not a solution.. – G. Stergiopoulos Feb 11 '22 at 23:06
  • @G.Stergiopoulos could you derive the correct value for `START_VALUE` based on the number of lines of your file? why would that not work? – Tadhg McDonald-Jensen Feb 11 '22 at 23:15
  • @G.Stergiopoulos so there's no way for you to *know* about when this will fail, and to handle it? Because if you are terminating gracefully, you can just pickle your `itertools.combinations` object. If you want to be able to start over, you'd have to periodically do that in *case* something catastrophic happens – juanpa.arrivillaga Feb 11 '22 at 23:28
  • @TadhgMcDonald-Jensen because they don't want to re-iterate, I guess. Which is possible because you can copy/serialize an `itertools.combinations` object and pick up where you left off, but the main issue is that the op wants to handle things like power outages... – juanpa.arrivillaga Feb 11 '22 at 23:28
  • @juanpa.arrivillaga ok but fast-forwarding `combinations` iterator with `islice` isn't going to take nearly as much time as restarting the entire loop, It just comes down to knowing the right index to pass to `islice` or fast-forwarding it by value like in my posted answer. Am I missing something because your repeated "don't want to re-iterate" is confusing me. – Tadhg McDonald-Jensen Feb 11 '22 at 23:42
  • @TadhgMcDonald-Jensen 's answer below suits perfectly, since it allows me to manipulate starting values, which is exactly what I needed. – G. Stergiopoulos Feb 12 '22 at 15:38

1 Answers1

0

as a somewhat dirty hack, iter(f, sentinel) produces an iterator that repeatedly calls the given function until it returns the given sentinel value, so if you knew the last finished value you could pass combos.__next__ and the last given value to iter and exhaust that iterator to skip to the next viable item:

import itertools, collections
all_combos = iter(map(set, itertools.combinations(range(1, 128), 3)))
LAST_RECORDED_COMBO = {121, 98, 45}
ITER_WE_WANT_TO_SKIP = iter(all_combos.__next__, LAST_RECORDED_COMBO)
# taken right from https://stackoverflow.com/questions/36763084/how-to-efficiently-exhaust-an-iterator-in-a-oneliner
collections.deque(ITER_WE_WANT_TO_SKIP, maxlen=0)

for combo in all_combos: # all_combos has now skipped forward
    print(combo) # original loop etc.

The above is little more than than just looping through the combos stopping when it sees the last one used like this:

for c in all_combos:
    if c == LAST_RECORDED_COMBO:
        break
for combo in all_combos: # all_combos has now skipped forward
    print(combo) # original loop etc.

But optimized so it can run entirely on the C layer much like islice to help performance.

I used map(set, ..) here because the sentinel value must be equal which means the ordering of a list or tuple would matter for it to stop properly, otherwise this has really bad feedback since it will just exhaust the entire iterator without a clear indication of why.

Tadhg McDonald-Jensen
  • 20,699
  • 5
  • 35
  • 59