How to automate jupyter notebook execution on aws?

Question

I got a task to complete where I need to automate Jupyter notebook execution on AWS. I'm totally new to AWS environment so don't have any idea how to do it efficiently. Things I need to do are the following -

Need REST API(s) to start and stop Jupyter notebook execution on AWS.
Need to send parameters to the notebook while calling using API.

What are the AWS components I need, to perform the above task?

import boto3,time
 
emr = boto3.client(
    'emr',
    region_name='us-west-1'
)
 
 
 
start_resp = emr.start_notebook_execution(
    EditorId='e-40AC8ZO6EGGCPJ4DLO48KGGGI',
    RelativePath='boto3_demo.ipynb',
    ExecutionEngine={'Id':'j-1HYZS6JQKV11Q'},
    ServiceRole='EMR_Notebooks_DefaultRole'
)
 
execution_id = start_resp["NotebookExecutionId"]
print(execution_id)
print("\n")
 
 
describe_response = emr.describe_notebook_execution(NotebookExecutionId=execution_id)
 
print(describe_response)
print("\n")
 
 
 
list_response = emr.list_notebook_executions()
print("Existing notebook executions:\n")
for execution in list_response['NotebookExecutions']:
    print(execution)
    print("\n")
 
 
 
print("Sleeping for 5 sec...")
time.sleep(5)
 
print("Stop execution " + execution_id)
emr.stop_notebook_execution(NotebookExecutionId=execution_id)
describe_response = emr.describe_notebook_execution(NotebookExecutionId=execution_id)
print(describe_response)
print("\n")

I have written a lambda function which I'm invoking using API gateway. In the lambda function, I'm using Boto3 to start notebook execution but the problem is every time I'm calling this function the notebook is entering into the start state and immediately going to stop state and not entering into the running state(logs from cloudwatch). — user22, Oct 30 '21 at 19:01
This is the code written in the lambda function which I'm calling using API Gateway. — user22, Oct 30 '21 at 19:51

score 1 · Answer 1 · answered Feb 18 '22 at 21:37

Here is an experimental repository that creates a Jupyter Extension in SageMaker Studio to automate running your notebooks. It has been designed for trial use and there's no guarantee of SageMaker support.

https://github.com/aws-samples/sagemaker-run-notebook

I work at AWS and my opinions are my own.

How to automate jupyter notebook execution on aws?

1 Answers1