AWS Lambda error: not all arguments converted during string formatting

Question

I have a python lambda code which iterates over a loop and starts 17 instances of a reusable glue job by passing run time arguments. I am using boto3 function which is start_job_run() to start the job. Below is the snippet:

    job_keys = {}
    for job_key_name in jobs:
        try:
            job_keys[job_key_name] = glue_client.start_job_run(JobName = glueJobName, Arguments = {'--Path': CONFIG_PATH, '--JobKeyName': job_key_name}) <--- Error is happening here
            logger.info(f'## STARTED GLUE JOB: {glueJobName}-{job_key_name}')
            logger.info(f'## GLUE JOB RUN ID: {job_keys[job_key_name]["JobRunId"]}')
            logger.info(f'## SOURCE: {source}')
        except Exception as e:
            logger.error('ERROR DETAILS', e)
            logger.error(job_key_name)
            pass

The code runs fine and starts 17 executions of the glue job. But there are few intermittent instances when the lambda fails with the error not all arguments converted during string formatting. When this happens the particular job instance does not start.

Note: The max concurrency of the glue job is set to 17 and no. of workers is 2.

i think is coming from logger, please refer this : https://stackoverflow.com/questions/12843099/python-logging-typeerror-not-all-arguments-converted-during-string-formatting/12843568 — Abhishek, Oct 05 '21 at 06:55
Thank you Abhishek. I will look into it. However, this issue is intermittent and is happening in the middle and not in the first iteration itself. If this was a logging issue, shouldn't it be happening every time?But your point is right, f string is not completely compatible with logging (Ref: https://stackoverflow.com/questions/54367975/python-3-7-logging-f-strings-vs). Will check.Another point to note is that the iteration where the error occurs that particular job is not executed. Ideally, that job should start atleast as the logger comes after the start_glue_job_run() — Aatisha, Oct 05 '21 at 07:33

score 1 · Answer 1 · answered Oct 05 '21 at 12:52

1

Your error is actually occurring in the Exception itself, but the error is masking another.

logger.error('ERROR DETAILS', e)

when receiving a botocore.ClientError exception will fail at this point with the mentioned error as logger.error has no idea how to process a boto3 ClientError object into a string (there are a lot of additional items on the ClientException object)

If you add the following:

from botocore.exceptions import ClientError
... [ your code ]
except ClientError as e:
    ...[handle the client error specifically]
except Exception as e:
    ...[the rest of your exception handling]

it should allow you to start to see the Client Error issue that is occurring - as it is intermittent, my guess would be that you are running up against glue limits for too many concurrent executions

answered Oct 05 '21 at 12:52

lynkfox

2,003
1
8
16

Thanks for the suggestion. Looks like this is the cause. After handling the client error exception I am now able to see the actual error. By any chance is there a possiblity that the f'string formatting logger.info() could be an issue? – Aatisha Oct 06 '21 at 05:42
generally no. f'string' formating is generally safe - mostly because if an object can't properly be converted to a string you just end up with [Type: XYZ] instead of an error. Where logger.error('msg', e) tries to pick up some more things from the exception that ClientErrors don't have hooks for. If that helped solve your problem, dont forget to mark it as the answer! – lynkfox Oct 07 '21 at 12:04

AWS Lambda error: not all arguments converted during string formatting

1 Answers1