2

I am trying to use a post-startup script to create a Vertex AI User Managed Notebook whose Jupyter Lab has a dedicated virtual environment and corresponding computing kernel when first launched. I have had success creating the instance and then, as a second manual step from within the Jupyter Lab > Terminal, running a bash script like so:

#!/bin/bash
cd /home/jupyter
mkdir -p env
cd env
python3 -m venv envName --system-site-packages
source envName/bin/activate
envName/bin/python3 -m pip install --upgrade pip
python -m ipykernel install --user --name=envName
pip3 install geemap --user 
pip3 install earthengine-api --user 
pip3 install ipyleaflet --user 
pip3 install folium --user 
pip3 install voila --user 
pip3 install jupyterlab_widgets
deactivate
jupyter labextension install --no-build @jupyter-widgets/jupyterlab-manager jupyter-leaflet
jupyter lab build --dev-build=False --minimize=False
jupyter labextension enable @jupyter-widgets/jupyterlab-manager

However, I have not had luck using this code as a post-startup script (being supplied through the console creation tools, as opposed to command line, thus far). When I open Jupyter Lab and look at the relevant structures, I find that there is no environment or kernel. Could someone please provide a working example that accomplishes my aim, or otherwise describe the order of build steps that one would follow?

user6135514
  • 135
  • 9
  • Have to tried supplying the start up script when creating a user-managed notebook from the [console](https://cloud.google.com/vertex-ai/docs/workbench/user-managed/create-new#set-instance-properties)? – Prajna Rai T Sep 09 '22 at 09:47
  • @PrajnaRaiT I have tried both console and command line, and neither has worked for me. – user6135514 Sep 12 '22 at 09:17

2 Answers2

2

Post startup scripts run as root. When you run:

python -m ipykernel install --user --name=envName

Notebook is using current user which is root vs when you use Terminal, which is running as jupyter user.

Option 1) Have 2 scripts:

  • Script A. Contents specified in original post. Example: gs://newsml-us-central1/so73649262.sh
  • Script B. Downloads script and execute it as jupyter. Example: gs://newsml-us-central1/so1.sh and use it as post-startup script.
#!/bin/bash

set -x

gsutil cp gs://newsml-us-central1/so73649262.sh /home/jupyter
chown jupyter /home/jupyter/so73649262.sh
chmod a+x /home/jupyter/so73649262.sh
su -c '/home/jupyter/so73649262.sh' jupyter

Option 2) Create a file in bash using EOF. Write the contents into a single file and execute it as mentioned above.

gogasca
  • 9,283
  • 6
  • 80
  • 125
  • `@gogasca` Many thanks! Method 1 generally works (haven't yet tested 2). On start-up, I see a single exposed kernel which contains all of the correct modules. How would I adjust the workflow to include both a default kernel (as if I had not used the post-startup script), and the custom environment, as if run from the Jupyter Lab > Terminal? Also, when run as a post-startup script, the custom kernel lacks the name provided in the script. Is this an easy fix? – user6135514 Sep 12 '22 at 11:05
  • Nevermind on the naming, I realized my error. However, while I got this to work once, when I repeat the process, I am getting an error. Script B copies Script A to the instance, so I can see exactly what I used on the working Notebook instance. When I try to use the working code to build another instance, it fails with the errors: `Failed to load model class 'LeafletMapModel' from module 'jupyter-leaflet'` `Error: No version of module jupyter-leaflet is registered`. I also get prompted to rebuild jupyter lab, which fails. Any thoughts on what is happening here? – user6135514 Sep 12 '22 at 12:28
  • 1
    Since default Jupyter version is 3.X based on this are you sure you need the 3 last lines? https://ipywidgets.readthedocs.io/en/7.x/user_install.html#installation https://github.com/jupyter-widgets/ipyleaflet#installation ? – gogasca Sep 13 '22 at 05:36
  • I am not certain. The idea, based on tests run from within Jupyter Lab > Terminal, was to force reinstallation of potentially absent or disabled extensions (which should turn them on by default) and to programmatically enable the extensions so the end user didn't need to click on the puzzle piece icon. It might redundant or unnecessary, and I will keep testing to see what can be removed. With that said, I think I have sorted out the major road blocks and will post additional insights in an edit or answer. – user6135514 Sep 13 '22 at 10:26
  • Installing new packages can cause pip or conda to downgrade the jupyter_server package from 3.X when it solves for the new environment. This would then need those last 3 lines. A constraint like `conda update jupyterlab_server>=2.19.0` removes this possibility. – intotecho Feb 23 '23 at 05:03
1

This is being posted as support context for the accepted solution from @gogasca.

@gogasca's suggestion (I'm using Option 1) works great, if you are patient. Through many attempts, I discovered that inconsistent behavior was based on timing of access. Using Option 1, the User Managed Notebook appears available for use in Vertex AI Workbench (green check and clickable "OPEN JUPYTERLAB" link) before the installation script(s) have finished.

If you open the Notebook too soon, you will find two things: (1) you will be prompted for a recommended Jupyter Lab build, for instance:

Build Recommended
JupyterLab build is suggested:
@jupyter-widgets/jupyterlab-manager changed from file:../extensions/jupyter-widgets-jupyterlab-manager-3.1.1.tgz to file:../extensions/jupyter-widgets-jupyterlab-manager-5.0.3.tgz

and (2) while the custom environment/kernel is present and accessible, if you try to use ipyleaflet or ipywidget tools, you will see one of several JavaScript errors, depending on how quickly you try to use the kernel, relative to the build that is (apparently) continuing to take place in the background: Error displaying widget: model not found, and/or a broken page icon with a JavaScript error, that, if clicked, will show you something like:

[Open Browser Console for more detailed log - Double click to close this message]
Failed to load model class 'LeafletMapModel' from module 'jupyter-leaflet'
Error: No version of module jupyter-leaflet is registered
at f.loadClass (https://someURL.notebooks.googleusercontent.com/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/134.bcbea9feb6e7c4da7530.js?v=bcbea9feb6e7c4da7530:1:74856)
at f.loadModelClass (https://someURL.notebooks.googleusercontent.com/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/150.3e1e5adfd821b9b96340.js?v=3e1e5adfd821b9b96340:1:10729)
at f._make_model (https://someURL.notebooks.googleusercontent.com/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/150.3e1e5adfd821b9b96340.js?v=3e1e5adfd821b9b96340:1:7517)
at f.new_model (https://someURL.notebooks.googleusercontent.com/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/150.3e1e5adfd821b9b96340.js?v=3e1e5adfd821b9b96340:1:5137)
at https://someURL.notebooks.googleusercontent.com/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/150.3e1e5adfd821b9b96340.js?v=3e1e5adfd821b9b96340:1:6385
at Array.map ()
at f._loadFromKernel (https://someURL.notebooks.googleusercontent.com/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/150.3e1e5adfd821b9b96340.js?v=3e1e5adfd821b9b96340:1:6278)
at async f.restoreWidgets (https://someURL.notebooks.googleusercontent.com/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/134.bcbea9feb6e7c4da7530.js?v=bcbea9feb6e7c4da7530:1:77764)

The solution here is to keep waiting. In my demo script, I transfer a file at the end of the build process. If I wait long enough for this file to actually appear in the Instance directories, the recommendation for a rebuild is absent and the extensions work properly.

user6135514
  • 135
  • 9
  • Today was looking into an issue in which we are updating OS after creation, may be related as we need to wait around 5 minutes after all patches are installed. (UI is accesible during this updates) – gogasca Sep 14 '22 at 04:52
  • I have returned to this after several days, and now find that the scripts that I had working no longer work. The environment is created and populated with the correct modules, but the ipyleaflet/ipywidget components now fail to work. Any ideas @gogasca? – user6135514 Sep 20 '22 at 11:55
  • 1
    New image m96 has issues with ipywidgets: https://github.com/jupyter-widgets/ipywidgets/issues/3598 which seems to be related to: https://github.com/jupyter-widgets/ipywidgets/issues/3600#issuecomment-1256697827 – gogasca Sep 24 '22 at 08:16