Best practice integrating pyenv virtualenv conda and ASE

Question

I am using a desktop with Linux (specifically Pop_OS! 20.04, which is analogous to ubuntu).

I need to run ASE which is python package for atomistic simulations. This can be installed in several ways (apt, pip, conda). For my purpose I need to install it with conda (to be as consistent as possible with the community). Since I do not like the way conda is integrated in bash, I decided to use pyenv as Virtual Environment manager.

Accordingly to answer 1 on another stackoverflow question, it is better not to install conda through pyenv. So in my simple mind firstly I tried this (CASE A):

I create a virtual environment with pyenv;
I install conda using the proper script and now pyenv
within the environment and using conda I install ASE

$ pyenv virtualenv 3.9.0 caseA
(caseA) $ which python  
/home/fex/.pyenv/shims/python
(caseA) $ python -V
Python 3.9.0
(caseA) $ bash Anaconda3-2020.07-Linux-x86_64.sh
# I did not include anaconda in the bashrc
# To activate conda eval "$(/home/user/anaconda3/bin/conda shell.YOUR_SHELL_NAME hook)" or "source <path to conda>/bin/activate"
# conda init
(baseA) $ source /home/fex/anaconda3/bin/activate
(base) (caseA) $

This double enviroment is quite interesting, and I think it should not happen. Yet I could not install ASE with conda. So I gaveconda init:

(base)(caseA) $ conda init
(base)(caseA) $ source ~/.bashrc
(base) $ conda install -c conda-forge/label/cf202003 ase
(base) $ ase test
...
fio/oi.py: TypeError: Object does not appear to be a file-like object
========== Summary ==========
Number of tests   363
Passes:           260
Failures:           0
Errors:             1
Skipped:          102
=============================
Test suite failed!
Time elapsed: 42.3 s

As far as I understand this method works, but bypass pyenv and I end up just using a conda environment which is not what i want in first place.

CASE B: This is not crucial. Once I created the environment I tried to install conda with pyenv, but I could not manage to use it within the pyenv environment.

CASE C: I solve the problem in the following way:

Installed conda with pyenv
create a pyenv virtual environment that relies on conda and not a specific version of python

$ pyenv virtualenv miniconda3.X.Y fooenv

This works for me, but I am using the python included in conda. This is not a problem, but since I am not completely grasping the full technicality here I wanted to share my doubts.

Another reason because I wrote here is that ASE is a pretty popular standard in the field of DFT (Density Functional Theory) which at date is one of the most cited topic in chemistry/physics fields. Yet the writing of manuals and guides does not keep up with the fast pace of python.

Extra info

$ pyenv versions
  system
  2.7.18
* 3.9.0 (set by PYENV_VERSION environment variable)
  3.9.0/envs/caseA
  3.9.0/envs/caseB
  3.9.0/envs/coppie
  3.9.0/envs/foo
  caseA
  caseB
  ccase
  coppie
  foo
  miniconda3-latest
  miniconda3-latest/envs/ccase # I created this with pyenv virtualenv miniconda3-latest ccase ccase stands for case C
$ pyenv -v
pyenv 1.2.21
$ which python
/home/fex/.pyenv/shims/python
$ python -V
Python 3.9.0
 pyenv activate ccase
pyenv-virtualenv: prompt changing will be removed from future release. configure `export PYENV_VIRTUALENV_DISABLE_PROMPT=1' to simulate the behavior.
(ccase) $ pip install pytest
(ccase) $ ase test
[...]
====================================================================== 1290 passed, 233 skipped, 1 xfailed, 5 warnings in 193.06s (0:03:13)

This seems awfully complex and brittle, why not just use Conda directly? — AMC, Oct 29 '20 at 01:04
That's a good point. Mostly because I do not like how it interacts with my bash (you can blame my technical skills for that too) and because it seems more appropriate to have an environment manager for applications outside of anaconda. — Cippo1987, Oct 29 '20 at 10:49
_I do not like how it interacts with my bash_ Can you elaborate? _it seems more appropriate to have an environment manager for applications outside of anaconda_ What do you mean? — AMC, Oct 30 '20 at 00:31
I have to anticipate one aspect. I do research with software written in FORTRAN for distributed calculation over HPC. Specifically I do not even touch that code, and I do not care too much about code at all since I work on the results of such code, that as far as i am concerned could be the result of a rand function. 1. Since I am not expert in many ways on how conda works, in the past, I had my bashrc completely fuc***up by conda (my fault). This included slowing down any login on the terminal. 2. Some packages are not available with conda, and I need to use them outside — Cippo1987, Nov 01 '20 at 13:47
I know that these are most excuses, but I work in a field where many procedures are done with old code which is nor maintained nor updated (e.g. python interface for blender to import atomic coordinate as 3D objects). 3. Basically a friend of mine who works in IT suggested me pyenv which worked incredibly fine, until I needed to conda as I explained in the initial questions. The bottom line is that nowadays they require me to use a little of python despite not giving me the tools or the time to learn and use it properly. — Cippo1987, Nov 01 '20 at 13:53
Ah, makes sense. #1 is unfortunate, I can understand not wanting to take the risk of something breaking. What do you mean by _slowing down any login on the terminal_? As for #2, you can use tools other than Conda, like pip, inside of a Conda environment. — AMC, Nov 03 '20 at 01:47
In this particular situation, I think using pyenv with Conda makes things needlessly complex, and most likely increases the chance that something will break. _The bottom line is that nowadays they require me to use a little of python despite not giving me the tools or the time to learn and use it properly._ Yeah, that sucks :/ Have you got it working for now? Related post, might be useful: https://stackoverflow.com/q/58044214. — AMC, Nov 03 '20 at 01:52
@AMC Basically I think the configuration of conda was xxxxed-up and since by default it got inserted in the .bashrc, everytime I was opening a shell it took 1 second. Same situation I have seen on different HPC (not managed by me). 2) Indeed. Anyway I wanted to decouple conda from my machine somehow. AT_amd2 You mean installing conda inside? Actually instsally conda with pyenv and creating directly a conda pyenv-enviroment so far works as a charm including anaconda-navigator. Anyway, a bit of autodafe' I mostly blind trusted a friend on pyenv. — Cippo1987, Nov 03 '20 at 13:13
_Basically I think the configuration of conda was xxxxed-up and since by default it got inserted in the .bashrc, everytime I was opening a shell it took 1 second._ Oh, that might be the fact that by default the base environment is activated automatically. — AMC, Nov 03 '20 at 23:41
_You mean installing conda inside?_ Which part are you responding to? The _you can use tools other than Conda, like pip, inside of a Conda environment._ ? If so, what I meant is that you should be able to treat the Python installation inside a Conda environment like any other. My apologies if my comment was unclear, let me know if it still isn't quite there yet. — AMC, Nov 03 '20 at 23:42
@amc _Oh, that might be the fact that by default the base environment is activated automatically._ Yes that was the reason, but it was unberable, and if you have a sporadic use, it is annoying to activate it each time. You get the point in the second message. Since I do not need to mess with python basically I do not create a conda virtual environment, but I create a virtual one where conda does all it needs. Also this allow me to save space i guess, with respect to create separate environment inside conda (not sure if it links the packages) — Cippo1987, Nov 04 '20 at 13:05
_Yes that was the reason, but it was unberable, and if you have a sporadic use, it is annoying to activate it each time._ Do you mean *deactivate* it each time? If so, you can change a setting so that it doesn't activate automatically (https://stackoverflow.com/q/54429210). _Since I do not need to mess with python basically I do not create a conda virtual environment, but I create a virtual one where conda does all it needs._ What do you mean by _mess with python_ ? You're still creating environments either way, no? — AMC, Nov 06 '20 at 03:36
_Also this allow me to save space i guess, with respect to create separate environment inside conda (not sure if it links the packages)_ Conda should avoid duplicating/redownloading the same package multiple times. The amount of storage used should be negligible anyway, unless you're facing some severe constraints. — AMC, Nov 06 '20 at 03:37
> You're still creating environments either way, no? I basically do not use python. It is mostly used by 3rd party application. another example Avogadro2. I imagine that conda also uses linking. So you are right. — Cippo1987, Nov 06 '20 at 14:04

Cippo1987 · Answer 1 · 2020-11-08T12:50:09.123

At this stage I managed to install ASE with conda. I did this inside a pyenv environment which calls directly miniconda instead of python.

$ pyenv virtualenv miniconda3.X.Y fooenv
$ pyenv activate fooend
 (fooenv) $ conda install -c conda-forge/label/cf202003 ase # This installs ASE 
 (fooenv) $ pip install pytest # In this version of ase `ase test` does not work
 (fooenv) $ ase test # This test is longer and more complete that the once achieve with conda without pyenv

Best practice integrating pyenv virtualenv conda and ASE

1 Answers1