How to use Conda kernels while executing jupyter notebook from lambda function

Question

How can I execute jupyter notebook using pre-installed conda kernels from lambda function instead of using the default kernel?

Currently, I'm using the following code snippet to execute jupyter notebook from the lambda function.

import time
import boto3
import logging
import requests
import websocket

def lambda_handler(event, context):
    sm_client = boto3.client('sagemaker')
    notebook_instance_name = 'Test-Instance'
    url = sm_client.create_presigned_notebook_instance_url \
                (NotebookInstanceName=notebook_instance_name)['AuthorizedUrl']
 
    url_tokens = url.split('/')
    http_proto = url_tokens[0]
    http_hn = url_tokens[2].split('?')[0].split('#')[0]
    
    s = requests.Session()
    r = s.get(url)
    cookies = "; ".join(key + "=" + value for key, value in s.cookies.items())
    
    ws = websocket.create_connection(
        "wss://{}/terminals/websocket/5".format(http_hn),
        cookie=cookies,
        host=http_hn,
        origin=http_proto + "//" + http_hn,
        header = [
            "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36"
        ]
    )
    
    ws.send("""[ "stdin", "jupyter nbconvert --execute --to notebook --inplace /home/ec2-user/SageMaker/test.ipynb --ExecutePreprocessor.kernel_name=python3 --ExecutePreprocessor.timeout=1500\\r" ]""")

When I execute jupyter notebook from lambda, it uses the default python kernel available for the Sagemaker instance which has python3.7 installed but if I execute my notebook from sagemaker instance than that uses conda python3.8

Following are the list of available kernels when executing from sagemaker instance:

R
Sparkmagic (PySpark)
Sparkmagic (Spark)
Sparkmagic (SparkR)
conda_amazonei_mxnet_p36
conda_amazonei_pytorch_latest_p37
conda_amazonei_tensorflow2_p36
conda_mxnet_p37
conda_python3
conda_pytorch_p38
conda_tensorflow2_p38

When I try to execute it from lambda function then only available kernels are:

python3
ir

This is what I tried: I created a custom kernel using conda and tried to use that while executing from lambda function, but it didn't work. In fact, if I give anything else other than python3 in kernel name while executing from lambda function, it doesn't execute the notebook.

score 0 · Answer 1 · answered Nov 15 '22 at 00:34

0

See this github issue

tl;dr - activate the conda environment you like to execute on, and then use the jupyter nbconvert command. jupyter kernelspec list should show you the available kernels (kernel_name=python3 should work within the conda_python3 environment)

answered Nov 15 '22 at 00:34

durga_sury

869
4
6

How can I activate a Conda environment on sagemaker from lambda script and then use one of the available kernels? – Muhammad Azeem Nov 15 '22 at 06:25
`kernel_name=python3` works within the default python kernel available in the sagemaker instance instead of `conda_python3`. – Muhammad Azeem Nov 15 '22 at 06:26
`jupyter kernelspec list` outputs only two kernels: `python3` and `ir` but not `conda_python3` as mentioned in the question – Muhammad Azeem Nov 15 '22 at 06:27
@MuhammadAzeem - Activate the `conda_python3` environment, and then run in the `python3` kernel. – durga_sury Nov 15 '22 at 16:21

How to use Conda kernels while executing jupyter notebook from lambda function

1 Answers1