2

I am using a modified form of the grouper recipe provided by the python docs:

from itertools import chain, islice
def grouper(iterable, n):
    iterable = iter(iterable)
    while True:
        peek = next(iterable)
        yield chain((peek,), islice(iterable, n - 1))

This seems to work fine. I can do something like:

>>> x = bytearray(range(16))
>>> [int.from_bytes(n, 'big') for n in grouper(x, 4)]
[66051, 67438087, 134810123, 202182159]

However, when I run the exact same code in IPython, I get a DeprecationWarning:

In [1]: from itertools import chain, islice
   ...: def grouper(iterable, n):
   ...:      iterable = iter(iterable)
   ...:      while True:
   ...:          peek = next(iterable)
   ...:          yield chain((peek,), islice(iterable, n - 1))

In [2]: x = bytearray(range(16))

In [3]: [int.from_bytes(n, 'big') for n in grouper(x, 4)]
__main__:1: DeprecationWarning: generator 'grouper' raised StopIteration
Out[3]: [66051, 67438087, 134810123, 202182159]

Where is the warning coming from and why do I not see it in the regular Python console? What can I do to make the warning go away?

I am using Python 3.6.2 and IPython 6.1.0

Mad Physicist
  • 107,652
  • 25
  • 181
  • 264

3 Answers3

4

This is a change to Python that is being gradually phased in between Python 3.5 and Python 3.7. The details are explained in PEP 479, but I'll try to give a quick overview.

The issue is StopIteration exceptions that are leaking out of generator functions. It might seem like that's OK, since raising a StopIteration is the signal an iterator is finished. But it can cause problems with refactoring generators. Here's an example showing the trouble:

Say you had this generator function (which works fine in Python versions prior to 3.5, where it started emitting warnings):

def gen():
    yield 1
    yield 2
    if True:
        raise StopIteration
    yield 3
    yield 4

Since the condition of the if is truthy, the generator will stop after yielding two values (not yielding 3 or 4). However, what if you try to refactor the middle part of the function? If you move the part from yield 2 to yield 3 into a helper generator, you'll see an issue:

def gen():
    yield 1
    yield from refactored_helper()
    yield 4

def refactored_helper():
    yield 2
    if True:
        raise StopIteration
    yield 3

In this version, 3 will be skipped, but 4 will still be yielded. That's because the yield from ate the StopIteration that was raised in the helper generator function. It assumed that only the helper generator should be stopped, and so the outer generator kept running.

To fix this, the Python devs decided to change how generators work. Starting in Python 3.7, a StopIteration exception that leaks out of a generator function will be changed by the interpreter to a RuntimeError exception. If you want to exit a generator normally, you need to use return. Furthermore, you can now return a value from a generator function. The value will be contained in the StopIteration exception that gets raised by the generator machinery, and a yield from expression will evaluate to the returned value.

So the generators above could properly be refactored to:

def gen():
    yield 1
    if yield from refactored_helper():
        return
    yield 4

def refactored_helper():
    yield 2
    if True:
        return True
    yield 3
    # like a normal function, a generator returns None if it reaches the end of the code

If you want to write future-compatible code now, you should put from __future__ import generator_stop at the top of your module. Then you need to track down places where you're leaking StopIteration exceptions and wrap them with try and except logic. For the code in your question:

from __future__ import generator_stop

from itertools import chain, islice

def grouper(iterable, n):
    iterable = iter(iterable)
    while True:
        try:
            peek = next(iterable)
        except StopIteration:
            return
        yield chain((peek,), islice(iterable, n - 1))
Blckknght
  • 100,903
  • 11
  • 120
  • 169
  • This is the answer I was looking for. Thanks. – Mad Physicist Aug 10 '17 at 08:54
  • What is the purpose of the import in this case? It does not appear to add any feature that does not already exist in 3.6. – Mad Physicist Aug 10 '17 at 15:10
  • I am asking because I tried your version and it seems to work just fine without the import. In case you were wondering, BTW, https://stackoverflow.com/a/23926929/2988730 – Mad Physicist Aug 10 '17 at 15:42
  • 1
    You're right that the `__future__` import isn't necessary in the last version of the code. However, it would break the earlier versions (they'd raise `RuntimeError`s). The idea is to add the future import, see what breaks in your existing code, then fix it to be forwards compatible. – Blckknght Aug 10 '17 at 18:20
  • Clear and helpful. Thanks! – PROgram52bc Nov 21 '20 at 05:56
1

The deprecation warning is thrown because of an upcoming non-backwards compatible change in the Python language which will be active from version 3.7 on and is documented in PEP-479.

The most important parts:

Abstract

This PEP proposes a change to generators: when StopIteration is raised inside a generator, it is replaced it with RuntimeError . (More precisely, this happens when the exception is about to bubble out of the generator's stack frame.) Because the change is backwards incompatible, the feature is initially introduced using a __future__ statement.

and

Transition plan

Python 3.5: Enable new semantics under __future__ import; silent deprecation warning if StopIteration bubbles out of a generator not under __future__ import. Python 3.6: Non-silent deprecation warning. Python 3.7: Enable new semantics everywhere.

The documentation you have linked should be updated.

Klaus D.
  • 13,874
  • 5
  • 41
  • 48
0

Deprecation warning is thrown when a module will be deprecated in future.For now,you re fine to use it. Just include the following to your code to not see these warnings:

import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.simplefilter(action='ignore', category=DeprecationWarning)
cs95
  • 379,657
  • 97
  • 704
  • 746
raul
  • 631
  • 2
  • 10
  • 23