Python Virtual Environments Confusion

Question

I have been learning data science using python for about a year now. I have become quite comfortable with the syntax and model creation. I have exclusively used Google Colab just due to how convenient it is and I love the notebook style. However, one thing I do not understand is the environment stuff. Although I use Colab, I do have python and anaconda on my machine and have installed various packages using the exact following format: pip install (package name). When I open my terminal, the first line is lead with (base) and when I check the Environments tab in anaconda navigator, it appears as though I installed all of these packages into a base environment named base (root)? Is that right? If so, what would my environment's name be then? What is a base environment compared to a venv?

The reason I am asking is because if I ever decide to use an IDE in the future, I would need to set my environment to be able to run packages, correct?

Just for fun I want to try using R and its reticulate package that allows python use in R. As stated in the answer to this question, I need to set my virtual environment before I can use python in R. Would my virtual environment be base (root)?

I'm a complete noob about all of this environment stuff. Again, I just opened my terminal and typed pip install (package name) for all packages I've installed. Thanks for any help in advance.

Answer 1

So from your description, it sounds like your default Python installation on your computer is through Anaconda. If that's the case, base is actually going to be the name of the conda virtual environment that you're using.

Virtual environments can be tricky, so I'll walk you through what I usually do here.

First, you can always check which Python installation you're currently using by using the which command on Mac/Linux, or if you're using Windows the command will probably be where (if you're on Windows, this answer might be helpful: equivalent of 'which' in Windows.)

(base) ➜  ~ which python
/Users/steven/miniconda3/bin/python

From the above, you can see that my default Python is through Miniconda, which is just a small version of Anaconda.

This means that when you use pip to install packages, those are getting installed into this base conda environment. And, by the way, you can use the which command with pip as well, just to double-check that you're using the version of pip that's in your current environment:

(base) ➜  ~ which pip
/Users/steven/miniconda3/bin/pip

If you want to see the list of packages currently installed, you can do pip freeze, or conda env export. Both pip and conda are package managers, and if you're using an Anaconda Python installation then you can (generally) use either to install packages into your virtual environment.

(Quick side note: "virtual environments" are a general concept that can be implemented in different ways. Both conda and virtualenv are ways to use virtual environments in Python. I'm also a data scientist, and I use conda for all of my virtual environments.)

If you want to create a new virtual environment using conda, it's very straightforward. First, you can create the environment and install some packages right away, like pandas and matplotlib. Then you can activate that environment, check your version of python, and then deactivate it.

(base) ➜  ~ conda create -n my-new-environment pandas matplotlib
(base) ➜  ~ which python
/Users/steven/miniconda3/bin/python
(base) ➜  ~ conda activate my-new-environment
(my-new-environment) ➜  ~ which python
/Users/steven/miniconda3/envs/my-new-environment/bin/python
(my-new-environment) ➜  ~ conda deactivate
(base) ➜  ~ which python
/Users/steven/miniconda3/bin/python

And, if you want to see which conda virtual environments you currently have available, you can run conda env list.

Here's the documentation for conda environments, which I reference all the time: https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html

I hope this is helpful!