Error not getting captured in except block

Question

I am trying to capture the error in Python for the below error but I am unable to do so:

from app_store_scraper import AppStore
import requests
import logging

logger = logging.getLogger()

try:
    minecraft = AppStore(country='sk', app_name='telekom')
    minecraft.review(how_many=20)
    reviews = (minecraft.reviews)
    print(reviews)
except AttributeError:
    logging.error("Please check the appname")
except requests.exceptions.ConnectionError:
    logging.error("Connection error1")
except requests.TooManyRedirects:
    logging.error("Connection error2")
except requests.RequestException:
    logging.error("Connection error3")
except Exception as e:
   logging.error("xxxxxxxxxxxxxxxxxxxxxxx")

Error:

 Something went wrong: HTTPSConnectionPool(host='amp-api.apps.apple.com', port=443): Max retries exceeded with url: /v1/catalog/sk/apps/1106471260/reviews?l=en-GB&offset=0&limit=20&platform=web&additionalPlatforms=appletv%2Cipad%2Ciphone%2Cmac (Caused by ResponseError('too many 404 error responses'))

But it's not getting captured in any of the except blocks.

Note: I am using apple app scraper library in Python to scrape data.

Does this answer your question? https://stackoverflow.com/q/23013220/14739759 — anurag, Feb 09 '21 at 15:07
@anurag i am trying to capture the error for exception handling. — Debashish Das, Feb 09 '21 at 15:10
When running your code, I am getting: `2021-02-09 15:09:18,202 [ERROR] root - Please check the appname` How do I reproduce the scenario? — anurag, Feb 09 '21 at 15:12
The appname is telekom and the issue is happening intermittently. — Debashish Das, Feb 09 '21 at 15:13
I am basically trying to error handling there are two error I am getting one is this one and the other is " 'latin-1' codec can't encode character '\u016f' in position 31: ordinal not in range(256)" for the app country="cz", app_name="Můj T-Mobile" and both are not getting captured You can try to capture this error if you want to if you are not able to replicate — Debashish Das, Feb 09 '21 at 15:15
See, the error is not a Python error/exception that is raised to be caught. The logger is simply recording it and displaying it. So we might have to redirect `stderr` or change settings in the logger — anurag, Feb 09 '21 at 15:18
Ok can you help me with some sample code for redirect in such scenerios — Debashish Das, Feb 09 '21 at 15:21
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/228473/discussion-between-anurag-and-debashish-das). — anurag, Feb 09 '21 at 15:23

anurag · Answer 1 · 2021-02-09T18:34:06.127

The basic idea is to capture the logger output emitted from within the app_store_scraper library (the logger defined by their Base class) and raise a manual exception for retry error:

from app_store_scraper import AppStore
import logging
from io import StringIO
import urllib3
import sys


tmp_stderr = StringIO()

log_stream = StringIO()
log_handler = logging.StreamHandler(stream=log_stream)
log_handler.setLevel(logging.ERROR)
loggerBase = logging.getLogger("Base")
base_handlers = [oh for oh in  loggerBase.handlers]

class ModifyLogger():
    def __init__(self, stder, tmpstrm):
        self.stder = stder
        self.tmpstrm = tmpstrm

    def __enter__(self):
        #logging.disable(logging.ERROR)
        sys.stderr = self.tmpstrm   # redirecting stderr
        
        # Removing existing handlers for "Base" logger and adding our own
        for oh in base_handlers:
            loggerBase.removeHandler(oh)
        loggerBase.addHandler(log_handler)
        loggerBase.setLevel(logging.ERROR)

    def __exit__(self, exit_type, exit_value, exit_traceback):
        #logging.disable(logging.NOTSET)
        # Removing our handler for "Base" logger and adding originals back
        loggerBase.removeHandler(log_handler)
        for oh in base_handlers:
            loggerBase.addHandler(oh)
        
        sys.stderr = self.stder   # restoring stderr


with ModifyLogger(sys.stderr, tmp_stderr):      # a context manager
    try:
        minecraft = AppStore(country='sk', app_name='telekom')
        minecraft.review(how_many=20)
        reviews = (minecraft.reviews)
        print(reviews)

        logs = log_stream.getvalue()
        if len(logs.split('\n'))==2 and logs.find('HTTPSConnectionPool', 0, 100) and logs.find('Max retries exceeded', 0, -1):
            # Extracting the URL
            url1 = logs.split('=')[1].split(',')[0].strip("'").strip()
            url2 = logs.split(':')[3].split('(')[0].strip()
            # Raising a manual exception for maximum retries
            raise urllib3.exceptions.MaxRetryError(pool='HTTPSConnectionPool', url=url1+url2, reason='Max Retries exceeded')

    except urllib3.exceptions.MaxRetryError as e:
        print('\"{}\" while accessing URL: {}'.format(e.reason, e.url))

The output from above code:

[]
"Max Retries exceeded" while accessing URL: amp-api.apps.apple.com/v1/catalog/sk/apps/1106471260/reviews?l=en-GB&offset=0&limit=20&platform=web&additionalPlatforms=appletv%2Cipad%2Ciphone%2Cmac

if we change the `app_name='dregs jjs'` we can test for a positive result! — anurag, Feb 09 '21 at 18:37
For the other errors, please provide more details for handling — anurag, Feb 09 '21 at 18:40
Hi Anurag ...i want to set a generic exception if the string "Something went wrong:" is encountered irrespective of the type of error as all error that will come wil come with this suffix in apple pp scrapper — Debashish Das, Feb 12 '21 at 07:09
Something went wrong: 'latin-1' codec can't encode character '\u016f' in position 31: ordinal not in range(256) — Debashish Das, Feb 12 '21 at 07:19
can you provide multiple lines of your logs so that I can test code around that? — anurag, Feb 12 '21 at 16:58
"ERROR] Base - Something went wrong: 'latin-1' codec can't encode character '\u016f' in position 31: ordinal not in range(256)" "Something went wrong: HTTPSConnectionPool(host='amp-api.apps.apple.com', port=443): Max retries exceeded with url: /v1/catalog/sk/apps/1106471260/reviews?l=en-GB&offset=0&limit=20&platform=web&additionalPlatforms=appletv%2Cipad%2Ciphone%2Cmac (Caused by ResponseError('too many 404 error responses'))" These are two errors i had got till now — Debashish Das, Feb 15 '21 at 09:36

Error not getting captured in except block

1 Answers1