AWS Lambda function with python, Boto3, and QuickSight SDK, getting error message: InvalidParameterValueException/Invalid ResourceId

Question

I recieved an interesting error message and (may) have an idea what's going on but need some guidance on the best way to resolve it.

I have a CloudFormation Stack with a Lambda Function that is creating some AIM policies and permissions when generating a QuickSight Data Source. A portion of the function here:

    response1 = quicksight_client.create_data_source(
    AwsAccountId='*******',
    DataSourceId= csv_file_1, 
    Name= csv_file_1,
    Type='S3',
    DataSourceParameters={
        'S3Parameters': {
            'ManifestFileLocation': {
                'Bucket': '******-quicksight-manifest',
                'Key': '********.json'
            }
        },
    },
    Permissions=[
        {
            'Principal': 'arn:aws:quicksight:us-east-1:*******:user/default/********', 
            'Actions': [
                "quicksight:DescribeDataSet",
                "quicksight:DescribeDataSetPermissions",
                "quicksight:PassDataSet",
                "quicksight:DescribeIngestion",
                "quicksight:ListIngestions",
                "quicksight:UpdateDataSet",
                "quicksight:DeleteDataSet",
                "quicksight:CreateIngestion",
                "quicksight:CancelIngestion",
                "quicksight:UpdateDataSetPermissions"
            ]
        },
    ],

    SslProperties={
        'DisableSsl': True
    },
    Tags=[
        {
            'Key': '******',
            'Value': 'Test'
        },
    ]
)

The error from the logs indicates


[ERROR] InvalidParameterValueException: An error occurred (InvalidParameterValueException) when calling the CreateDataSource operation: Invalid ResourceId. Check if it exceeds the length limit or contains invalid characters.
Traceback (most recent call last):
  File "/var/task/handler.py", line 100, in featureengineering
    response1 = quicksight_client.create_data_source(
  File "/var/runtime/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 676, in _make_api_call
    raise error_class(parsed_response, operation_name)

Doing a search found a similar situation with a solution, but with JavaScript.

What would be the best way, or to implement that in Python? Or, are there some other options? such as placing time.sleep() somewhere within the function, and if so, where?

Also, I'm assuming that allowing AWS to replicate the role through the regions would solve this issue, but I might be wrong and going down a blind ally, and the issue is something unrelated.

Thank you.

Edit/Update

I realized something that is pretty obvious but didn't see it because I fixated on other situations which were similar, but the IAM resources that I am creating for that bucket are being created through CloudFormation in the Yaml file, and independent of the Lambda Function or when it's triggered.

As such, I I redeployed the stack (including the IAM resources) and then left it alone for about 25 minutes before triggering the function, and still received that error message, so I don't think it is a result of the AIM role not propagating through AWS in time.

But, I'm pretty new to Boto3, and I might have made a mistake in instantiating the client. I have quicksight_client = boto3.client("quicksight") but should it be quicksight_client = boto3.resource("quicksight") instead?

Additional Edit

So, I attempted to try it with boto3.resource("quicksight") but no, that's not going to work. That resource does not exist, and got an error message.

[ERROR] ResourceNotExistsError: The 'quicksight' resource does not exist.
The available resources are:
   - cloudformation
   - cloudwatch
   - dynamodb
   - ec2
   - glacier
   - iam
   - opsworks
   - s3
   - sns
   - sqs

Consider using a boto3.client('quicksight') instead of a resource for 'quicksight'
Traceback (most recent call last):
  File "/var/lang/lib/python3.8/imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "/var/lang/lib/python3.8/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 702, in _load
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/var/task/handler.py", line 19, in <module>
    quicksight_client = boto3.resource("quicksight")
  File "/var/runtime/boto3/__init__.py", line 100, in resource
    return _get_default_session().resource(*args, **kwargs)
  File "/var/runtime/boto3/session.py", line 346, in resource
    raise ResourceNotExistsError(service_name, available,

Potentially still could be a timing issue, perhaps.

Not sure if this is helpful. But, it looks like you have a trailing `,` when there shouldn't be under `DataSourceParameters` — TheoNeUpKID, Nov 07 '20 at 21:28
Ok, yeah. You mean the one with the S3Parameters? if if's the bracket associated with the `S3Parameters{}` that might be it because the API was expecting multiple DataSourceParameters, but I'm only giving it one (S3Parameter), and that would make that `,` unnecessary. — JLuu, Nov 07 '20 at 21:56
Maybe try boto3 IAM's `get_waiter()` function? https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/iam.html#IAM.Client.get_waiter — Frank, Nov 07 '20 at 22:32
@Frank potentially yeah. I like that I wouldn't have to inject more dependencies such as time, and it would potentially be more efficent than setting than setting an arbitrary value that stalls the function. I'll have to look for examples of how it's used. Do you have any examples of how that syntax would be implemented? — JLuu, Nov 08 '20 at 02:19
Found this as an example of wait for iam creation: https://stackoverflow.com/questions/55406664/boto3-iam-waiter-isnt-waiting — Frank, Nov 08 '20 at 17:42
I just did an edit, and I don't think it's an IAM propogation-timing issue. Should I be using `Quicksight_client = boto3.resource("quicksight")` instead of `Quicksight_client = boto3.client("quicksight")`? — JLuu, Nov 09 '20 at 15:39

score 0 · Answer 1 · answered Mar 24 '21 at 00:15

A little later here, but I ran into the same error and only found this unanswered question. Hoping to save others some time and pain.

After a process of elimination I have found the following:

The error came down to this line:

DataSourceId= csv_file_1,

Though mine was using some other string identifier. After looking at Data Sources created through the console, I found that AWS is using UUIDs here and other formats are not accepted. I decided to use the native uuid package for this:

import uuid
...
DataSourceId=str(uuid.uuid4())

This is also true for the other Quicksight resources that require an Id attribute as a parameter such as Data Sets, Dashboards, etc.

Additionally, you will want to make sure you have an applicable permission set. The ones shown in OP would throw a new error. Fortunately in those cases the error provides a solution. In my case I used the following permission set:

'Actions': [
  'quicksight:UpdateDataSourcePermissions',
  'quicksight:DescribeDataSource',
  'quicksight:DescribeDataSourcePermissions',
  'quicksight:PassDataSource',
  'quicksight:UpdateDataSource',
  'quicksight:DeleteDataSource',
]

Cheers!

AWS Lambda function with python, Boto3, and QuickSight SDK, getting error message: InvalidParameterValueException/Invalid ResourceId

1 Answers1