How do we freeze an entire dependency tree/chain from an already working opam install?
I have a bunch of coq projects installed via opam right now. I'd like to figure out what commit they are using for their current install. I want to know this -- ideally automatically -- such that the opam pkgs/projects and their dependencies are pinned/fixed/frozen so that future opam installs/pins on the never break on them. For example if I knew the commit I could construct the opam switch need and install it with:
eval $(opam env --switch=coq-8.10 --set-switch)
opam pin add coq-metalib https://github.com/plclub/metalib/commits/104fd9efbfd048b7df25dbac7b971f41e8e67897
how do I do this? at the very least I'd like to be able to get commits like 104fd9efbfd048b7df25dbac7b971f41e8e67897 from opam installs that are already working.
How to automatically determine the git commit to make a permanent opam pin add url.git#commit install?
If I have a working set-up with opam. How do I get all the git commits for each opam project/package I have so to make the current install permanent? e.g. some projects it seems in coq get over written in the OPAM official repo so I want to lock those with git commits. So far manually checking them works:
eval $(opam env --switch=coq-8.10 --set-switch)
opam pin add -y coq-metalib git+https://github.com/plclub/metalib.git#104fd9efbfd048b7df25dbac7b971f41e8e67897
but I have a massive list of opam packages working right now and would like to make their install robust with the git commits. e.g. all these work for now:
# --- Install all Opam Dependencies: 1. create opam switch needed 2. then install all opam dependencies & projs
opam list
# - Create the 8.10.2 switch
opam switch create coq-8.10 4.07.1
eval $(opam env --switch=coq-8.10 --set-switch)
opam pin add -y coq 8.10.2
# - Install dependency packages for 8.10
opam repo add coq-extra-dev https://coq.inria.fr/opam/extra-dev
# We don't need it in all opam switches due to incompatabilities: Run `opam repository add <coq-proj> --all-switches|--set-default' to use it in all existing switches, or in newly created switches, respectively. cmd: opam repository add coq-extra-dev --all-switches
opam repo add coq-released https://coq.inria.fr/opam/released
opam repo add psl-opam-repository https://github.com/uds-psl/psl-opam-repository.git
opam install -y coq-serapi
opam install -y coq-struct-tact
opam install -y coq-inf-seq-ext
opam install -y coq-smpl
opam install -y coq-int-map
opam install -y coq-pocklington
opam install -y coq-mathcomp-ssreflect coq-mathcomp-bigenough coq-mathcomp-algebra
opam install -y coq-fcsl-pcm
opam install -y coq-list-string
opam install -y coq-error-handlers
opam install -y coq-function-ninjas
opam install -y coq-algebra
opam install -y coq-zorns-lemma
opam pin -y add menhir 20190626
# coq-equations seems to rely on ocamlfind for it's build, but doesn't
# list it as a dependency, so opam sometimes tries to install
# coq-equations before ocamlfind. Splitting this into a separate
# install call prevents that. https://stackoverflow.com/questions/75452026/how-do-i-install-ocamlfind-first-properly-before-other-opam-packages-without-roo, untested for now
opam install -y ocamlfind
opam install -y coq-equations coq-metacoq coq-metacoq-checker coq-metacoq-template
# lin-alg-8.10 needs opam switch coq-8.10
git submodule add -f --name coq-projects/lin-alg-8.10 git@github.com:HazardousPeach/lin-alg-8.10.git coq-projects/lin-alg
git submodule update --init coq-projects/lin-alg
(cd coq-projects/lin-alg && make "$@" && make install)
# to confirm it installed look for lin-alg: https://github.com/UCSD-PL/proverbot9001/issues/81, for now you can confirm by trying to install it again and it all looks alright
#opam list
#opam list | grep lin-alg-8.10
# Install the psl base-library from source
mkdir -p deps
git clone -b coq-8.10 git@github.com:uds-psl/base-library.git deps/base-library
(cd deps/base-library && make "$@" && make install)
git clone git@github.com:davidnowak/bellantonicook.git deps/bellantonicook
(cd deps/bellantonicook && make "$@" && make install)
opam list | grep base-library
# -- Get cheerios, req to have old versions work in opam: https://github.com/uwplse/cheerios/issues/17
eval $(opam env --switch=coq-8.10 --set-switch)
# opam install might give issues since it gets the most recent version from the official OPAM repository
#opam -y install coq-cheerios
#opam install -y coq-verdi
# use opam pin since pin is created to install specific version (e.g. from git, local, etc.)
opam pin add coq-cheerios git+https://github.com/uwplse/cheerios.git#9c7f66e57b91f706d70afa8ed99d64ed98ab367
#opam pin add coq-cheerios https://github.com/uwplse/cheerios.git\#9c7f66e57b91f706d70afa8ed99d64ed98ab367d
#opam pin add coq-verdi https://github.com/uwplse/verdi/tree/f3ef8d77afcac495c0864de119e83b25d294e8bb
opam pin add coq-verdi git+https://github.com/uwplse/verdi.git#f3ef8d77afcac495c0864de119e83b25d294e8bb
# use opam pin since pin is created to install specific version (e.g. from git, local, etc.)
Context: installing proverbot9001 coq-projects dependencies that have their OPAM repo code overwritten by original authors
I have this issue were I try to install an opam package (coq project/pkg in my specific case) that I have downloaded/cloned via a git submodule and when I do try to opam install it it fails. It fails, although I have cloned the right version of the coq project/pkg. The source of failure is something I've been told here because it is imho very hard to infer/deduce that from the error message:
(iit_synthesis) brando9~ $ eval $(opam env --switch=coq-8.10 --set-switch)
(iit_synthesis) brando9~ $ opam install -y coq-cheerios
[ERROR] Package conflict!
* Missing dependency:
- coq >= 8.14
not available because the package is pinned to version 8.10.2
No solution found, exiting
it says a newer version of coq is needed but it's very confusing why that is the case. In python there is a setup.py
or a requirements.txt
file that makes all of this a little bit more clear (dependency management is always a pain afaik :( ).
I am wondering, is there such a file in opam where I can specify for the coq-projects I am using which version of the dependencies I need so I don't have to go back and fix everything? e.g. in python I'd do:
pip3 freeze > requirements.txt # Python3
pip freeze > requirements.txt # Python2
can I do this in ocaml? Ideally automatically. Even more ideally looping through all the coq-projs in my current repo under the folder coq-projects/
and create such a freeze of the dependencies for future installs? Once that is there how would I install the downloaded/cloned coq proj with the specified forzen dependencies?
Why can't we just extract the deps from a list like pip does? It's linear once a working installation is set up
I don't understand something about opam vs pip. Once a working installation is set up -- it's not a chain or recursive structure of installations. It's simply a list of installations that can be frozen with pip. Is that not possible with opam? If not why not and why is it so hard to fix?
Useful links
related discussion to solve this issue:
- trying to get coq-cheerios & verdi to work by getting the right commits for them: https://github.com/uwplse/cheerios/issues/17 , trying to get support for cheerios for coq 8.10: https://github.com/uwplse/cheerios/issues/12
- this is a related question I asked when I thought it was my cloning of the coq-projects I was going to build How does one git submodule add a specific commit and have it be recorded in the .modules files? but before I even build them I need to get their dependencies installed! e.g. via opam. Some proverbot installed from source :( and Idk how to fix those yet e.g. metalib: https://github.com/UCSD-PL/proverbot9001/issues/82
- I might have an issue with coq-serapi too: https://github.com/UCSD-PL/proverbot9001/issues/78
- when I thought creating my own forks would work https://github.com/UCSD-PL/proverbot9001/issues/83 but likely just getting the opam install/pin to work with the commit is better e.g.
opam pin add coq-metalib https://github.com/plclub/metalib/commits/104fd9efbfd048b7df25dbac7b971f41e8e67897
. - cross at ocaml discuss: https://discuss.ocaml.org/t/how-does-one-pin-freeze-a-version-of-the-dependencies-of-an-opam-project-package-and-then-install-the-project-with-such-specified-dependencies/11424
- another discussion when running a giant set of opam installs: https://github.com/UCSD-PL/proverbot9001/issues/55
- related & likely culprit of issues; is it true that OPAM allows none addative updates to their official repository of packages? https://discuss.ocaml.org/t/is-it-true-that-the-official-opam-repository-allows-non-additive-updates-to-the-projects-packages-pushed-to-it/11432