Multi-user Conda Workflow

Question

Firstly, to mention what my goal is: To have a shared environment structure for 2+ users to access & use.

Current setup: I have a Windows Server 2016 with full admin access that's connected to the companies active directory. I've been using Python/Conda for a couple of years now but only for myself, I've never had to share any code to run under the same conditions.

I have tried various solutions trying to set this up & resulted in reinstalling anaconda/miniconda over & over for a fresh start however I keep running into permission issues between users (both users are admins on the server. I've also made a group on the server and gave the various anaconda (installed in different locations during my attempts; C:\, C:\ProgramData\, D:).

How the ideal scenario would work is: I create a new environment in Conda which installs to a shared location between the users. Once I create the environment, me & the other users can activate it, install packages and both users can use whatever package the other installs.

Where we're at the moment is, I've created an environment in D:\envs. I've then asked the other user to install pandas in that environment (we've also tried vice versa), pandas installs successfully but then says Access Denied for when I try to use that package.

I'm completely open to starting fresh if I need to or try other suggestions on the best way of doing this.

Answer 1

In the top level directory for the conda environment you will need to set ACL list such that the AD group that both users belong to (or just both users by name, but really, that is a bad option) have full control over all files, folders, and subfolders/files.

You must also set the inheritance so that all files/folders/objects created under the top level directory inherit this permission.

It is best to do this BEFORE you create the new environment so you don't have to go through and reset all of the inheritances after you create the files. A good way to do this is to create a folder that will hold your environments (I am doing this example in the D-drive):

D:
mkdir D:\conda

Then set the permissions on conda for the Active Directory group PythonUsers so that all sub-folders/files inherit.

icacls D:\conda /grant:r PythonUsers:(OI)(CI)F /T

This command changes the permissions for D:\conda by:

• replacing granted permissions (/grant:r)
• for the AD group PythonUsers
• with full control (F at the end)
• Also, set the inheritance so it applies to this level's folder and files (OI)
• and all subfolders/files (CI)
• The /T traverses to all subfolders that already exist, which should be none, but it does hurt.

Now, when you (or any user in the PythonUsers AD group) creates/modifies files in the D:\conda or below, all other users in PythonUsers should have full access to it.

At this point you could set your conda config to use the new directory as the default location for new environments and packages via:

conda config --prepend envs_dirs D:\conda\envs
conda config --prepend pkgs_dirs D:\conda\pkgs

Then fire up a new environment as usual:

conda create -n test python=3.7 pandas

Answer 2

As an alternative to using a shared location for the conda environment, have you considered maintaining an environment file which defines what packages should be in the environment, and letting each user create their own environment locally using that file (conda env create -f path\to\environment.yml)?

The file can be stored in your version control system and if you want you can probably even automate updating the conda env if an update from the VCS changes the file, via a hook in your VCS client.

This sounds like a less fragile and more scalable way of doing things to me, and maintaining an environment file that defines your project's dependencies is good practice in any case.