7

My project supposed to run on cross platform environment (Mac, Win, Linux). I've created a conda env that manage our dependencies for an easy setup. I want to ensure that everyone that want to update the enn could do that, however when I try to export the env from linux to yml file, it couldnt be install properly on Win or Mac and vise versa.

I've already tried to do the regular stuff:

1. conda env export > env.yml
conda env create --name -f env.yml

2. conda env export --no-builds > env.yml

3. https://medium.com/@Amet13/building-a-cross-platform-python-installer-using-conda-constructor-f91b70d393

4. https://tech.zegami.com/conda-constructor-tutorial-make-your-python-code-easy-to-install-cross-platform-f0c1f3096ae4

5. https://github.com/ESSS/conda-devenv/blob/master/README.rst

non of the above give me the right answer... some of the tutorials I've attached might help, but I didn't succeed to implement them correctly, and they didn't contain some important information for finishing the tutorial properly.

for instance: Regarding 3/4 - It didn't explain how to create the yml file that should construct the env.

I understood that conda supposed to work on cross platform env... It would be great if someone could help me with that.

rambo
  • 121
  • 1
  • 3
  • What do you mean by " it couldnt be install properly on Win or Mac" were some dependencies just not available on windows? – FlyingTeller Sep 19 '19 at 11:46
  • Those libraries might not have an installation package on Win or Mac... but I don't care about that.. I want that when I generate the yml file it would put automatically the packages the could be install on win of mac... Now I'm doing it manually, and I think there should be better way. In addition, some of the packages do have a installation version on win or mac, but conda doesn't manage it properly. – rambo Sep 19 '19 at 12:11
  • I don't think that exporting envs to yml is supposed to work that way, it is meant to export and recreate an exact environment. Also, it would become highly confusing to have the "same" env on the three platforms, but they are actually not the same. – FlyingTeller Sep 19 '19 at 12:41
  • I tried also to use conda-devenv that generate a different yml for each platform, but it didn't work as well. On of the principles of anaconda is to be easy install in cross platform env, but I don't see it happens – rambo Sep 19 '19 at 13:10
  • 1
    Does this answer your question? [How to share conda environments across platforms](https://stackoverflow.com/questions/39280638/how-to-share-conda-environments-across-platforms) – neves Oct 23 '20 at 15:29

1 Answers1

9

Conda Envs are Not Inherently Cross-Platform

Sorry, but what you're asking for is simply not a thing. Conda can serialize an environment's package information to a YAML (great for reproducibility), but it can't guarantee that it will be cross-platform. In fact, many packages, especially ones with non-Python code, require different underlying build tools as dependencies, so what you're asking for will never be satisfied.

Explicit Specs Only

The closest you can get these days is to limit your environment.yaml to only include explicit specs that have gone into creating your environment by using the --from-history flag. This feature requires Conda v4.7.12 or later.

conda env export --from-history > environment.yaml

This will generate a YAML that only includes the packages that have been explicitly requested in the history of the env, e.g., if your history goes...

conda create -n foo python=3.7 numpy
conda install -n foo pandas scikit-learn

Then the result of conda env export -n foo --from-history will be something like

name: foo
channels:
  - defaults
dependencies:
  - python=3.7
  - numpy
  - pandas
  - scikit-learn
prefix: /your/conda/dir/envs/foo

This way, you can leave out all the other dependencies that may turn out to be platform-specific.

I'm Still Seeing a Ton of Packages?!

I've noticed that if one ever uses the --update-deps flag in an env, it adds every dependency to being an explicit spec. This is rather unfortunate. If this is the case, I'd suggest recreating the env using your legitimate specs and avoid that flag in the future. Searching through your command history might be useful in compiling that legitimate spec list.

merv
  • 67,214
  • 13
  • 180
  • 245
  • 1
    `--from-history` appears to be deprecated and no longer works. – rjurney Dec 09 '20 at 22:19
  • @rjurney where are you seeing that? I see no evidence of that in the codebase or issues. – merv Dec 09 '20 at 22:24
  • It only happens on OS X, where it is not available, despite using identical conda environments. The problem with this approach is that if more than one person adds a library or you add it from another machine, it will be missing from the tank file. – rjurney Dec 11 '20 at 00:03
  • @rjurney please file [an issue on the repository](https://github.com/conda/conda/issues), and make sure to include the documentation they request (e.g. a `conda list` output from the env with the issue). If you tag me (@mfansler) I will have a look. – merv Dec 11 '20 at 00:08
  • Hi, random question. Is there any downside to using --from-history? Or is --from-history a strictly safer and better way to share my environment? – arjunc12 Dec 24 '20 at 19:06
  • @arjunc12 downside would be that unless the user gave specific versions when requesting packages, the current versions won't be included. So, when recreating, one could end up with different versions. Also any dependencies won't have versions recorded, so there could be some changes in APIs that may not be properly recorded as breaking changes for the depending package. Generally, if you know you want to work cross platform, just maintain a YAML from the start. – merv Dec 24 '20 at 19:15
  • question. there is no to include package version with `--from-history` flag ? I don;t want to work cross platform but I want to include packages with their versions that I specifically chose and let conda figure out dependencies in the new environment. – moth Feb 17 '21 at 10:54
  • @alex `—from-history` includes exactly what the user previously specified, so if you originally told it a specific version, it will include that version. You can always simply edit the file to include whatever you’d like. – merv Feb 17 '21 at 13:34
  • 1
    @merv yes it does but it doesn't include the package version, that's what I was lookinf for. have the `--from-history` feature with the package version. – moth Feb 18 '21 at 02:41
  • Note that --from-history *doesn't* include packages explicitly installed via `pip install`. It means that projects that require pip on top of conda cannot be exported cross-platform using --from-history. – amka66 Dec 15 '21 at 21:14
  • This doesnt keep channels... – SumNeuron Feb 17 '23 at 22:04
  • 1
    @SumNeuron using *ad hoc* `--channel,-c` flags when installing will not be captured by `conda env export` (with or with the `--from-history` flag). Unfortunately, it only captures the current state of `conda config --show channels`, which is honestly terrible. A more reproducible workflow is to **work in a YAML first approach**: manually write a YAML with your specifications (and channels) and always use that to create and update your environment (i.e., no `conda {install,update,remove}` commands). – merv Feb 20 '23 at 08:44
  • @merv that is what I have done. However it is a bit wonky. If I install from the complete yaml file. (with all the channels) it doesn't work. If I install piecewise there is no issue. – SumNeuron Feb 20 '23 at 13:54
  • 1
    @SumNeuron sounds like a `channel_priority` issue. Your channels are not in a good order and/or you need `flexible` instead of `strict`. – merv Feb 20 '23 at 18:19