3

I've been trying to add one of my folders where I hold my python modules and, so far, I haven't been able to do it through AWS's terminal. The folder with the .py files is inside the main SageMaker folder, so I'm trying (I've also tried it with SageMaker/zds, which is the folder that holds the modules):

export PYTHONPATH="${PYTHONPATH}:SageMaker/"

After printing the directories of the PYTHONPATH through the terminal with python -c "import sys; print('\n'.join(sys.path))", I get that indeed my new path is included in the PYTHONPATH. However, when I try to import any module from any notebook (with from zds.module import * or from module import *), I get the error that the module doesn't exist. If I print the paths from the PYTHONPATH directly inside the notebook I no longer see the previously added path in the list.

Am I missing something basic here or is it not possible to add paths to the PYTHONPATH inside AWS SageMaker? For now, I'm having to use import sys, os sys.path.insert(0, os.path.abspath('..')) inside basically every notebook as a fix to the problem.

Philippe Fanaro
  • 6,148
  • 6
  • 38
  • 76
  • [Setting a default sys.path for a Notebook](https://stackoverflow.com/q/38237284/608639) – jww Mar 11 '19 at 02:51
  • From what I understand, he is basically doing what I do but within an external file? – Philippe Fanaro Mar 11 '19 at 11:07
  • Apparently, besides `PYTHONPATH`, there is a `JUPYTER_PATH`. However, changing it haven't had any local impact, essentially the same behavior as `PYTHONPATH`. – Philippe Fanaro Mar 11 '19 at 22:41
  • Please don't take this comment as condescending, I've been guilty myself a hundred times. Have you restarted the terminal after modifying the path? – Kerri Mar 12 '19 at 13:43
  • Yes, I did try restarting it. Did it work for you? (btw, I don't mind that kind of questioning at all, as long as it stays on the topic and not on presuming the other is ignorant.) – Philippe Fanaro Mar 15 '19 at 23:56

3 Answers3

2

Adding this to the lifecycle script worked for me

sudo -i <<'EOF'

touch /etc/profile.d/jupyter-env.sh
echo export PYTHONPATH="$PYTHONPATH:/home/ec2-user/SageMaker/repo-name/src" >> /etc/profile.d/jupyter-env.sh

EOF
Adam Mills
  • 7,719
  • 3
  • 31
  • 47
0

I have a folder with custom Python packages in my home area on SageMaker Studio. I'm running a Studio Notebook instance using the Data Science 3.0 image. The Studio home folder is mapped to /root inside the Notebook container. In my home folder, there's a hidden .ipython directory. I was able to permanently add my folder to the Python path by adding a startup script to ~/.ipython/profile_default/startup. In that directory, create a startup script something like this:

00-add-path.py

import sys
sys.path.append("/root/my-python-folder")

This method doesn't require a lifecycle configuration start-up script.

Scott
  • 2,764
  • 1
  • 16
  • 16
-2

Thanks for using Amazon SageMaker!

Copying from the https://docs.aws.amazon.com/sagemaker/latest/dg/notebook-lifecycle-config.html

Amazon SageMaker notebook instances use conda environments to implement different kernels for Jupyter notebooks. If you want to install packages that are available to one or more notebook kernels, enclose the commands to install the packages with conda environment commands that activate the conda environment that contains the kernel where you want to install the package For example, if you want to install a package only in for the python3 environment, use the following code:

# This will affect only the Jupyter kernel called "conda_python3".
source activate python3

# Replace myPackage with the name of the package you want to install.
pip install myPackage
# You can also perform "conda install" here as well.

source deactivate

If you do installation in above suggested way you should be able to import your package from the Notebook corresponding Kernel which you are using. Let us know if it doesn't help.

Pranav Chiplunkar
  • 163
  • 4
  • Hey, thanks for the help. But, from what I understand, your answer is regarding libraries that are available through `pip` or the `conda` command? My question was related to local libraries. Through `pip`, the libary's path is implicitly (I think) included in the `PYTHONPATH`, however, if you have local libraries, you have to include it manually, but it isn't working for me when I try it with Jupyter. – Philippe Fanaro Mar 22 '19 at 18:52