1

the situation is that sometimes a request does not load or gets stuck in Python, in case that happens or any error occurs, I would like to retry it "n" times and wait up to a maximum of 3 seconds for each one and in case the attempts are over tell me a message that f"Could not process {type_1} and {type_2}". Everything runs in parallel with concurrent.futures. Could you help me with that?

import Requests
import concurrent.futures
import json

data = [['PEN','USD'],['USD','EUR']]

def currency(element):

  type_1 =element[0]
  type_2 = element[1]

  s =  requests.Session()
  url = f'https://usa.visa.com/cmsapi/fx/rates?amount=1&fee=0&utcConvertedDate=07%2F26%2F2022&exchangedate=07%2F26%2F2022&fromCurr={type_1}&toCurr={type_2}'
  a = s.get(url)
  response = json.loads(a)
  value = response["convertedAmount"]

  return value

with concurrent.futures.ProcessPoolExecutor() as executor:
      results = executor.map(
                currency, data)
for value in results:
    print(value)
Dreku
  • 33
  • 6

1 Answers1

2

Your code is almost there. Here, I modified a few things:

from concurrent.futures import ThreadPoolExecutor
import time

import requests


def convert_currency(tup):
    from_currency, to_currency = tup
    url = (
        "https://usa.visa.com/cmsapi/fx/rates?amount=1&fee=0"
        "&utcConvertedDate=07%2F26%2F2022&exchangedate=07%2F26%2F2022&"
        f"fromCurr={from_currency}&toCurr={to_currency}"
    )
    session = requests.Session()
    for _ in range(3):
        try:
            response = session.get(url, timeout=3)
            if response.ok:
                return response.json()["convertedAmount"]
        except requests.exceptions.ConnectTimeout:
            time.sleep(3)
    return f"Could not process {from_currency} and {to_currency}"


data = [["VND", "XYZ"], ['PEN','USD'], ["ABC", "XYZ"], ['USD','EUR'], ["USD", "XXX"]]
with ThreadPoolExecutor() as executor:
    results = executor.map(convert_currency, data)
for value in results:
    print(value)

Notes

  • I retried 3 times (see the for loop)
  • Use timeout= to specify the time out (in seconds)
  • The .ok attribute will tell if the call was successful
  • No need to import json as the response object can JSON decode with the .json() method
  • You might experiment between ThreadPoolExecutor and ProcessPoolExecutor to see which one performs better
Hai Vu
  • 37,849
  • 11
  • 66
  • 93
  • It's amazing but is there any possibility to improve it by using retries = Retry() instead of a loop? – Dreku Aug 01 '22 at 15:32
  • Sometimes i had requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='usa.visa.com', port=443): Max retries exceeded with url: /cmsapi/fx/rates?amount=1&fee=0&utcConvertedDate=08%2F01%2F2022&exchangedate=08%2F01%2F2022&fromCurr=IDR&toCurr=MOP (Caused by ConnectTimeoutError(, 'Connection to usa.visa.com timed out. (connect timeout=2)')) – Dreku Aug 01 '22 at 17:03
  • What is `Retry()`? – Hai Vu Aug 02 '22 at 14:50
  • I have added `try/except` block to handle `requests.exceptions.ConnectTimeout` error. – Hai Vu Aug 02 '22 at 14:53
  • I saw something like this https://stackoverflow.com/questions/15431044/can-i-set-max-retries-for-requests-request . it seems to me that Retry retries it without a try except, that's new to me too – Dreku Aug 02 '22 at 19:31