1

I find myself needing to create very similar conda environments which, in terms of space is inefficient. I was wondering if there was a way to layer different environments in a "modular" way. For example, having a 'common' env with pandas, numpy, and other basic everyday needed libraries. And on top of that, activate other environments with other libraries, for example, streamlit to run a dashboard or one with deep learning libraries such as fastai or keras.

Some of these libraries like the ones used in deep learning take up a lot of space in the disc, so having 3 different projects with the same version of keras, just because they use a different version of some other library is something I'm trying to avoid.

I've read the conda documentation and I see there's a --stack option which at first seems to be what I'm looking for but, testing it out I cannot run the first's environment libraries once I activate the second one.

Am I doing something wrong or is this just something that is not possible with conda? And if not, is there any alternative to archiving it?

Thanks in advance

Tomas Ambrogi
  • 53
  • 5
  • 2
    It actually isn't space inefficient to create multiple environments with the same/similar packages, because conda is able to directly reuse one single version of the files and link them into environments as necessary. See: https://stackoverflow.com/questions/55566419/why-are-packages-installed-rather-than-just-linked-to-a-specific-environment Also, regarding this comment: "having 3 different projects with the same version of keras, just because they use a different version of some other library is something I'm trying to avoid." This is EXACTLY the point of environments :-) They keep... – darthbith Nov 02 '20 at 17:43
  • 1
    dependencies separate when there is a difference based on some of the packages you need to use. If you want to keep everything consistent, use a `environment.yaml` file with the build number specified, and only create environments from that. – darthbith Nov 02 '20 at 17:44
  • 2
    No, `--stack` can't do this. I've given [my best attempt at a thorough answer](https://stackoverflow.com/questions/55222405/can-conda-environment-inherit-base-packages/64384990#64384990) to the oldest variant of this question, but unfortunately I can't propose as dupe yet since it lacks an upvote. – merv Nov 02 '20 at 18:19
  • @darthbith I know that is exactly the point of environments haha that's why I'm using them :). When I said avoid I meant to be able to reuse the packages when necessary, I didn't know conda did that by default. Thanks for the reference! – Tomas Ambrogi Nov 03 '20 at 19:12
  • 1
    Hey @merv, thanks for the reference as well, the question darthbirth cited was answered by you so thanks again. Seems I was worrying for nothing since conda creates hardlinks to avoid multiple installations of the same package as you explain. Thanks! – Tomas Ambrogi Nov 03 '20 at 19:15
  • 1
    @darthbirth There is one big exception to that rule, however, in that it is only true for environments created by the same user. So if base is installed as root, and I `conda create --clone` it, I will get a complete copy of the base system, not links. If every user on a shared system does so, that is a lot of wasted disk space. – pavon Apr 01 '21 at 00:14

0 Answers0