38

I'm trying to implement requests retry in Python.
It works like charm with .get() requests, but a .post() request never retries, regardless of a status code. I'd like to use it with .post() requests.

My code:

from requests.packages.urllib3.util import Retry
from requests.adapters import HTTPAdapter
from requests import Session, exceptions

s = Session()
s.mount('http://', HTTPAdapter(max_retries=Retry(total=2, backoff_factor=1, status_forcelist=[ 500, 502, 503, 504, 521])))
r = s.get('http://httpstat.us/500')
r2 = s.post('http://httpstat.us/500')

So, the .get() requests do retry and the .post() ones do not.

What's wrong?

Stanislav Shabalin
  • 19,028
  • 3
  • 18
  • 18
  • 1
    Is it supposed to work like you expect? `GET` requests won't harm data, but multiple `POST` might. I haven't read through the requests API documentation, but it sounds reasonable if this is by design. – Jim Aho Feb 29 '16 at 16:38

2 Answers2

44

In urllib3 POST is not allowed as a retried method by default (since it can cause multiple inserts). You can force it though:

Retry(total=3, allowed_methods=frozenset(['GET', 'POST']))

See https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#urllib3.util.retry.Retry

Stanislav Shabalin
  • 19,028
  • 3
  • 18
  • 18
  • Could you explain the logic behind using `frozenset` in this snippet, please? – Amir Afianian Jan 05 '21 at 12:27
  • 1
    @AmirAfianian Check out linked documentation for Retry. The reason is to be type-consistent with default value of `allowed_methods=frozenset({'DELETE', 'GET', 'HEAD', 'OPTIONS', 'PUT', 'TRACE'})` – Stanislav Shabalin Feb 01 '21 at 17:10
3

You can use tenacity.

doc: https://tenacity.readthedocs.io/en/latest/

And you can log before or after

pip install tenacity

import logging

logging.basicConfig(stream=sys.stderr, level=logging.DEBUG)

logger = logging.getLogger(__name__)

@retry(stop=stop_after_attempt(3), before=before_log(logger, logging.DEBUG))
def post_something():
    # post
    raise MyException("Fail")
samuel161
  • 221
  • 3
  • 2