SageMaker Studio credentials errors

Question

I can no longer view projects, pipelines or create new note books. The complaint appears to be credential related.

Failed to launch app [None]. Error when retrieving credentials from container-role: Error retrieving metadata: Received non 200 response (400) from ECS metadata: ResourceNotFoundException

Has anyone run into this before? Nothing should have changed in the environment.

Additionally, when I view my user in SageMaker console, it doesn't have an arn listed under the Execution Role section:

Editing the user and re-saving it doesn't populate the role either. The role exists in IAM and it appears in the wizard when I edit the user details:

Do you have access to the sagemaker service? You could try to restart the server. Also you can click in file -> exit -> suspend all application and server and enter again, that would restart the instance of the server. — Jose, Jun 20 '23 at 16:01
I have access to the SageMaker console in AWS if that's what you mean? Where can I restart the service from within there? In SageMaker Studio when I go to File-Shutdown it throws a similar error "Error message: Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1. Args: undefined" — Ray Hogan, Jun 20 '23 at 16:04
Yes, I meant the console in AWS. I think you can do it in Domains -> User Profile, and if you click in the user you can check all the instances that are running and you can try to restart them or delete them (it won't delete your data, only the instance running). — Jose, Jun 20 '23 at 19:16

Warlax56 · Answer 1 · 2023-07-20T21:06:32.470

3

Creating a more intuitive UI based solution:

go to sage maker:
go to the affected domain:
go to the affected user profile
delete this.

then relaunch studio as you normally would. If you deleted the right server it should say that it's creating a new one. Your files will remain.

edited Jul 20 '23 at 21:06

answered Jul 13 '23 at 17:50

Warlax56

1,170
5
30

In my scenario, this didn't work. The only way to delete the server was through the cli. – Ray Hogan Jul 14 '23 at 18:29
For clarity: I personally did have this issue, and this approach did solve it. – Warlax56 Jul 19 '23 at 19:18

score 2 · Accepted Answer · answered Jun 21 '23 at 19:45

2

We resolved this issue by deleting the JupyterServer from the user profile and then re-launching Studio which will automatically create a new one (user data isn't affected as it is stored in EFS). One caveat is that we were unable to use the UI to delete the JupyterServer. The delete option was there but nothing happened when clicked, instead we had to use the CLI to remove it.

This had to be done on every user profile.

answered Jun 21 '23 at 19:45

Ray Hogan

319
3
12

1

I can confirm that it solved it for me too (it worked from UI). Thank you! – Giuseppe La Gualano Jun 22 '23 at 15:50
1

Oh my god thank you. We had 500 errors but otherwise the same and this fix worked after deleting the user's JupyterServer from the AWS Console. – KTamas Jun 28 '23 at 14:18
Errors like this make me want to move to colab. – Warlax56 Jul 13 '23 at 17:42

SageMaker Studio credentials errors

2 Answers2