0

I am a novice Conda user (~1 year of using; mostly Miniconda, FWIW) and I have occasionally run into issues trying to create a Conda environment (env) from a YAML file, seemingly because of specific file versions being listed. Does specifying specific versions as well as channels in a YAML make an environment less likely to solve in the future? Specifically, do channels that contain a specific version at the time of creating a YAML file ever wipe those versions from their package listing in the future?

Command:

conda env create -f snap.yml

Stdout:

Collecting package metadata (repodata.json): done
Solving environment: failed

ResolvePackageNotFound:

- markupsafe==1.1.1=py36h68aa20f_3
- ipykernel==5.5.0=py36h7b2dad6_1
- vc==14.2=h21ff451_1
- tornado==6.1=py36h68aa20f_1
- pip==21.0.1=py36haa95532_0
- mistune==0.8.4=py36h68aa20f_1003
- zeromq==4.3.3=h0e60522_3
- vs2015_runtime==14.27.29016=h5e58377_2
- m2w64-gcc-libs==5.3.0=7
- winpty==0.4.3=4
- m2w64-gcc-libgfortran==5.3.0=6
- m2w64-gcc-libs-core==5.3.0=7
- pandoc==2.11.4=h8ffe710_0
- libsodium==1.0.18=h8d14728_1
- pyzmq==22.0.3=py36hb0157bd_0
- msys2-conda-epoch==20160418=1
- jupyter_core==4.7.1=py36ha15d459_0
- wincertstore==0.2=py36h7fe50ca_0
- pywinpty==0.5.7=py36h9f0ad1d_1
- python==3.6.3=h3b118a2_4
- cffi==1.14.5=py36he58ceb7_0
- m2w64-libwinpthread-git==5.0.0.4634.697f757=2
- terminado==0.9.2=py36ha15d459_0
- pyrsistent==0.17.3=py36h68aa20f_2
- certifi==2020.12.5=py36ha15d459_1
- m2w64-gmp==6.1.0=2
- pywin32==300=py36h68aa20f_0
- setuptools==52.0.0=py36haa95532_0
- importlib-metadata==3.7.0=py36ha15d459_0
- notebook==6.2.0=py36ha15d459_0
- nbconvert==6.0.7=py36ha15d459_3
- argon2-cffi==20.1.0=py36h68aa20f_2

Contents of snap.yml:

name: snap
channels:

- conda-forge
- defaults
  dependencies:
- argon2-cffi=20.1.0=py36h68aa20f_2
- async_generator=1.10=py_0
- attrs=20.3.0=pyhd3deb0d_0
- backports=1.0=py_2
- backports.functools_lru_cache=1.6.1=py_0
- bleach=3.3.0=pyh44b312d_0
- certifi=2020.12.5=py36ha15d459_1
- cffi=1.14.5=py36he58ceb7_0
- colorama=0.4.4=pyh9f0ad1d_0
- decorator=4.4.2=py_0
- defusedxml=0.6.0=py_0
- entrypoints=0.3=pyhd8ed1ab_1003
- importlib-metadata=3.7.0=py36ha15d459_0
- importlib_metadata=3.7.0=hd8ed1ab_0
- ipykernel=5.5.0=py36h7b2dad6_1
- ipython=5.8.0=py36_1
- ipython_genutils=0.2.0=py_1
- jinja2=2.11.3=pyh44b312d_0
- jsonschema=3.2.0=py_2
- jupyter_client=6.1.11=pyhd8ed1ab_1
- jupyter_core=4.7.1=py36ha15d459_0
- jupyterlab_pygments=0.1.2=pyh9f0ad1d_0
- libsodium=1.0.18=h8d14728_1
- m2w64-gcc-libgfortran=5.3.0=6
- m2w64-gcc-libs=5.3.0=7
- m2w64-gcc-libs-core=5.3.0=7
- m2w64-gmp=6.1.0=2
- m2w64-libwinpthread-git=5.0.0.4634.697f757=2
- markupsafe=1.1.1=py36h68aa20f_3
- mistune=0.8.4=py36h68aa20f_1003
- msys2-conda-epoch=20160418=1
- nbclient=0.5.3=pyhd8ed1ab_0
- nbconvert=6.0.7=py36ha15d459_3
- nbformat=5.1.2
- nest-asyncio=1.4.3=pyhd8ed1ab_0
- notebook=6.2.0=py36ha15d459_0
- packaging=20.9=pyh44b312d_0
- pandoc=2.11.4=h8ffe710_0
- pandocfilters=1.4.2=py_1
- pickleshare=0.7.5=py_1003
- pip=21.0.1=py36haa95532_0
- prometheus_client=0.9.0=pyhd3deb0d_0
- prompt_toolkit=1.0.15=py_1
- pycparser=2.20=pyh9f0ad1d_2
- pygments=2.8.0=pyhd8ed1ab_0
- pyparsing=2.4.7=pyh9f0ad1d_0
- pyrsistent=0.17.3=py36h68aa20f_2
- python=3.6.3=h3b118a2_4
- python-dateutil=2.8.1=py_0
- python_abi=3.6=1_cp36m
- pywin32=300=py36h68aa20f_0
- pywinpty=0.5.7=py36h9f0ad1d_1
- pyzmq=22.0.3=py36hb0157bd_0
- send2trash=1.5.0=py_0
- setuptools=52.0.0=py36haa95532_0
- simplegeneric=0.8.1=py_1
- six=1.15.0=pyh9f0ad1d_0
- terminado=0.9.2=py36ha15d459_0
- testpath=0.4.4=py_0
- tornado=6.1=py36h68aa20f_1
- traitlets=4.3.3=py36h9f0ad1d_1
- typing_extensions=3.7.4.3=py_0
- vc=14.2=h21ff451_1
- vs2015_runtime=14.27.29016=h5e58377_2
- wcwidth=0.2.5=pyh9f0ad1d_2
- webencodings=0.5.1=py_1
- wheel=0.36.2=pyhd3eb1b0_0
- wincertstore=0.2=py36h7fe50ca_0
- winpty=0.4.3=4
- zeromq=4.3.3=h0e60522_3
- zipp=3.4.0=py_0
- pip:
  - absl-py==0.11.0
  - apricot-select==0.5.0
  - astor==0.8.1
  - cached-property==1.5.2
  - chardet==3.0.4
  - cleverhans==3.0.1
  - cloudpickle==1.6.0
  - common==0.1.2
  - cycler==0.10.0
  - efficientnet==1.0.0
  - future==0.18.2
  - gast==0.2.2
  - google-pasta==0.2.0
  - grpcio==1.36.0
  - h5py==2.10.0
  - hypothesis==5.5.4
  - idna==2.10
  - image-classifiers==1.0.0
  - imageio==2.9.0
  - joblib==1.0.1
  - keras==2.2.4
  - keras-applications==1.0.8
  - keras-preprocessing==1.1.2
  - keras-resnet==0.2.0
  - keras-unet==0.0.7
  - kiwisolver==1.3.1
  - llvmlite==0.35.0
  - markdown==3.3.4
  - matplotlib==3.3.4
  - mnist==0.2.2
  - networkx==2.5
  - nose==1.3.7
  - numba==0.52.0
  - numpy==1.19.5
  - opencv-python==4.2.0.32
  - opt-einsum==3.3.0
  - pandas==1.1.5
  - pillow==8.1.0
  - progressbar2==3.53.1
  - protobuf==3.15.3
  - pycodestyle==2.6.0
  - pydot==1.4.1
  - python-utils==2.3.0
  - pytz==2021.1
  - pywavelets==1.1.1
  - pyyaml==5.4.1
  - requests==2.23.0
  - scikit-image==0.16.2
  - scikit-learn==0.22.2
  - scipy==1.5.4
  - segmentation-models==1.0.1
  - sortedcontainers==2.3.0
  - tensorboard==1.15.0
  - tensorflow==1.15.2
  - tensorflow-estimator==1.15.1
  - tensorflow-probability==0.9.0
  - termcolor==1.1.0
  - tf-explain==0.2.1
  - tqdm==4.58.0
  - urllib3==1.25.11
  - werkzeug==1.0.1
  - wrapt==1.12.1
    prefix: C:\\Users\\zaki.conda\\envs\\snap

Upon reviewing similar questions, I also tried removing the suffixes after the second "=". This shortened the list quite a bit, but resulted in some packages still being listed:

Collecting package metadata (repodata.json): done
Solving environment: failed

ResolvePackageNotFound:

- winpty=0.4.3
- m2w64-gmp=6.1.0
- m2w64-gcc-libs=5.3.0
- vs2015_runtime=14.27.29016
- vc=14.2
- pywinpty=0.5.7
- m2w64-libwinpthread-git=5.0.0.4634.697f757
- pywin32=300
- m2w64-gcc-libs-core=5.3.0
- msys2-conda-epoch=20160418
- m2w64-gcc-libgfortran=5.3.0

EDIT: Okay I'm pretty sure this is just because this package wasn't made to work on linux, because all the missing packages seem windows-related. I'll wait to delete this post in case anyone is willing to expand on some of the original broader questions, even though this case is essentially solved.

merv
  • 67,214
  • 13
  • 180
  • 245
koestreich
  • 1
  • 2

1 Answers1

0

Yes and no.

The example YAML captures both versions and builds, which improves reproducibility on that specific platform (win-64 in this case). That is, it gives extra information to identify exactly which binary builds of packages were installed in the environment.

However, build information is specific to the platform, and so including it in the YAML can constrain the environment from being used across different platforms. Moreover, platforms often require specific supporting packages that are not available on other platforms. So, even leaving out builds and versions can still result in a YAML that is not portable across platforms.

The following questions have related information:

Reproducible Workflows

Recommendation: Do not rely on conda env export!

For anyone aiming to produce portable and reusable environment definitions, consider adopting a YAML-first workflow:

  • manually maintain a user-generated YAML file
  • only include end requirements
  • use only conda env create and conda env update commands to modify the environment instance.

An even better solution these days is conda-lock lockfiles. The lockfiles from conda-lock define solved enviornments, directly specify URLs to package tarballs, and support multiple platforms in a single file.

merv
  • 67,214
  • 13
  • 180
  • 245