18

Sorry for the length, this is a pretty intricate pipenv situation.

At my company we are using pipenv (with both Pipfile and Pipfile.lock) to control packages used on different engineers' laptops. This is even more important for us than for most teams because we're also using Zappa to deploy AWS Lambda code, and it apparently packages the dependencies directly from the deployer's laptop to deploy them. So if people's laptops aren't totally aligned in terms of dependencies, we can get different behavior in the cloud depending on who deployed it.

We have found that even after attempting to fully control dependencies with Pipfile and Pipfile.lock, we end up getting different Python packages on our different laptops, as shown by pip freeze and as indicated by errors in deployed code.

Here is the exact process that is showing differences between my laptop and my boss's (the Pipfile code I quote is on multiple lines but I'm condensing it to one line because I'm having trouble with SO formatting):

  1. At the very beginning, all we had was a Pipfile with packages specified with wildcards like [requires] python_version = "3.6" [packages] flask = "*". Also, we didn't have a Pipfile.lock, my boss (who was the first coder on this project) had always run --skip-lock
  2. To control things better, I started by upgrading our Pipfile to replace the wildcards with explicit versions and also make our Python version more specific, like [requires] python_version = "3.6.4" [packages] Flask = "==1.0.2". To do this, I got a copy of my boss's pip freeze output and copied the versions into the Pipfile where there was a name match with what was listed there (I skipped anything that didn't match because I assumed it was an upstream dependency and we weren't touching that yet). I committed this.
  3. We were still having problems, so we decided to start using Pipfile.lock to control upstream dependencies. So my boss created one by running pip install without --skip-lock for the first time, and committed that.
  4. I pulled the Pipfile.lock, deleted my environment with pipenv --rm and recreated it with pipenv install
  5. We both ran pip freeze and compared outputs, but we both still have a number of differences.

I suppose I can have my boss delete his pipenv environment and reinstall based on the committed Pipfile and Pipfile.lock, but since they are based on his pip freeze I would be a little surprised if that changed anything.

So I'm just wondering: is this behavior truly unexpected? I always thought the combination of pipenv, Pipfile, and Pipfile.lock would guarantee two people have the same packages, as long as every version is locked with ==[version]. Is there anything else we would need to do to get a very exact match?

If it's truly unexpected, the only other thing I can think is that maybe he hadn't run pipenv shell before his pip freeze, but I think he did because things lined up well against the Pipfiles.

Side note: I haven't converted our [dev-packages] in Pipfile to have versions because I'm not sure what that does and I'm assuming it's irrelevant. So those are still like pylint = "*"

ADDITIONAL INFO

Below is some additional info to respond to the comments... but first a couple of interesting things I noticed:

  • None of the differences in the first screenshot (for pip freeze diffs) are in the Pipfile.
  • It looks like my pip freeze output matches the Pipfile.lock contents, but my boss's doesn't. I think this might explain the differences, but it's a bit surprising that his pip freeze output wouldn't match the Pipfile.lock created by his own pipenv lock, unless the problem is that he ran pipenv lock from outside of pipenv shell.

To respond to the comments... Here is the first part of the diff between the pip freeze outputs (both from within pipenv shell) on my and my boss's laptops:

enter image description here

Here are some diffs in the Pipfile.lock between my and my boss's laptops. The Pipfile.lock was obtained by having him run pipenv lock (outside of pipenv shell although I assume that doesn't matter) and then committing that just now. I then pulled that, deleted my environment with pipenv --rm, ran pipenv install, and got the following differences with the Pipfile.lock that he had just committed. His version is on the left again.

These are all of the differences - one thing I don't get is why we have fewer differences here than with pip freeze. Our Pipfile is still the same between the two of us.

enter image description here

enter image description here

enter image description here

enter image description here

Stephen
  • 8,508
  • 12
  • 56
  • 96
  • with regards to the `dev-packages`, unless you're installing with `--dev`, that shouldn't have any impact. another question would be how are you creating your `Pipfile.lock`? are you using `pipenv lock` or just depending on the installs to add themselves to the lockfile? – wpercy May 02 '19 at 21:24
  • I would suggest having your boss run `pipenv lock` and committing the resulting `Pipfile.lock` – wpercy May 02 '19 at 21:24
  • What's an upstream dependency? A dependency of your dependencies? Also, can you be more specific about the 'differences' in versions? – Filip Dimitrovski May 02 '19 at 21:32
  • As Filip said, can you provide more details about the "differences" between running `pip freeze` on yours and your boss's computer? In addition, what does the differed packages look like in `Pipfile.lock` file? – Philip Tzou May 02 '19 at 21:49
  • Re: upstream dependency... yup I mean a dependency of my dependencies – Stephen May 06 '19 at 01:11
  • Will get the other answers tomorrow when I'm back at work – Stephen May 06 '19 at 01:11
  • Updated the question with some of the details requested here. I'm not sure how he was creating the lock before, but for this latest update he did `pipenv lock`. What would be the difference between that and automatically creating it? – Stephen May 06 '19 at 17:38
  • Key point from what I've found so far: It looks like my `pip freeze` output matches the `Pipfile.lock` contents, but my boss's doesn't. I think this might explain the differences, but it's a bit surprising that his `pip freeze` output wouldn't match the `Pipfile.lock` created by his own `pipenv lock`, unless the problem is that he ran `pipenv lock` from outside of `pipenv shell`. – Stephen May 06 '19 at 18:01
  • what's the timestamp on the pipfile.lock file that you're Boss checked in, compared to the file on his machine? Are they the same? – gregory May 06 '19 at 18:39
  • Sadly at this point we were trying other things so his local timestamp has since been updated – Stephen May 06 '19 at 19:11
  • Basically to sum things up, it seems my boss is generating `Pipfile.lock` output which is newer than what comes out with `pip freeze`. This happens regardless of whether `pipenv lock` is done within `pipenv shell` or not. – Stephen May 06 '19 at 19:22
  • As a side note/reflexion I was wondering if another management system like conda (the package management tool; not anaconda/miniconda) was better suited to your needs. Conda offers a full in-depth dependencies management. Some links https://jakevdp.github.io/blog/2016/08/25/conda-myths-and-misconceptions/ and https://stackoverflow.com/questions/39280638/how-to-share-conda-environments-across-platforms. Unfortunately _i don’t have an answer to that question, only a few links_ – LoneWanderer May 08 '19 at 09:23

2 Answers2

8

The only way to ensure you share the exact same environment is to synchronize with the same Pipfile.lock, with pipenv sync (optionally pipenv sync --dev).

Pipfile is a helper for humans, an intermediate in the Pipfile.lock creation, it does not ensure that dependencies are exactly the same.

pipenv install calls under the hood 2 pipenv function: lock and sync. pipenv lock will generate a Pipfile.lock from your Pipfile. Even with pinned version in Pipfile, it is possible to have different Pipfile.lock if they are generated at different moments because dependencies of the pinned packages may not be pinned (depending of the publisher). pipenv sync then install the exact packages found in the Pipfile.lock.

To directly install your environment from the dependencies in Pipfile.lock, you have to use pipenv --python 3.6 install --ignore-pipfile, otherwise Pipfile.lock will be regenerated from the Pipfile.

To easily solve your problem, fix a Pipfile.lock version (you can commit it if you use version control, but you do, of course ;), then both use pipenv sync.

Then keep the Pipfile.lock exactly the same as long as you work on minor version, bug fixes... and feel free to regenerate it to get up-to-date dependencies for major versions. In my project, almost all dependencies in the Pipfile are not pinned, and when we start a new major version we update the Pipfile.lock to try fresh dependency versions, test everything, sometimes pin a dependency to a previous version if the latest introduced backward incompatible changes, and we fix the Pipfile.lock until the next major version.

gaFF
  • 747
  • 3
  • 11
  • 1
    Thanks, this is really useful. I had no idea I had to run `sync` to actually use the `Pipenv.lock` for a deterministic install!! BTW, are you saying that `pipenv --python 3.6 install --ignore-pipfile` is the same thing as `pipenv sync`? – Stephen May 09 '19 at 21:16
  • I'm still not totally sure that this will solve the problem - because he did run install to create the lock, but the `Pipfile.lock` that he generated had newer packages than the packages that were shown when he ran `pip freeze`. It's strange because based on your description of what `pipenv install` does, it should have run a sync so that those two things would have matched... right? – Stephen May 09 '19 at 21:29
  • According to my tests, `pipenv install --ignore-pipfile` is similar to `pipenv sync`. – gaFF May 10 '19 at 08:53
  • `pip freeze` is not reliable, that's part of why `pipenv`and `Pipfile.lock` were created. In particular, the dependencies of your direct dependencies can be missed by `pip freeze`. I think that looking at the `site-package` of your virtual environment could be a better cross-check. You can access your virtual environment with `pipenv ---venv`, and `site-package` is in `lib/python3.X`. Try: `ls \`pipenv --venv\`/lib/python*/site-packages/`. – gaFF May 10 '19 at 09:00
  • 1
    In my mind, `pipenv` is not as intuitive and straightforward that it should be: `install` should be the current `sync` (install from `Pipfile.lock` only), `lock` should be the current `install` (regenerate `Pipfile.lock` and install from it), and generate the `Pipfile.lock` only should be non-direct, such as `pipenv lock --no-install`. – gaFF May 10 '19 at 09:04
  • 1
    Also be careful when you install a new package, because it will regenerate the `Pipfile.lock` entirely, instead of trying to append the package to the existing, pinned packages if there is no sub-dependencies conflict. – gaFF May 10 '19 at 09:12
1

pipenv install installs from Pipfile. Upstream dependencies may differ.

pipenv sync installs from Pipfile.lock. Nothing will differ.

That's my understanding from reading the command's help.

$ pipenv
Usage: pipenv [OPTIONS] COMMAND [ARGS]...

Commands:
  # ...
  install    Installs provided packages and adds them to Pipfile, or (if no
  # ...
  sync       Installs all packages specified in Pipfile.lock.
Eric Ihli
  • 1,722
  • 18
  • 30
  • 1
    Thanks, that's good to know about `pipenv sync`. Still, it doesn't address the issue of his generated `Pipfile.lock` apparently not matching his environment. See the comments on the question. – Stephen May 07 '19 at 18:42
  • I just re-read the question. Perhaps I'm still misreading it, but I think it does address the issue. If you two people do `pipenv install` at different times, it's very likely they will have different Pipfile.lock. `pipenv install` only looks at the `Pipfile`, not the `Pipfile.lock`. So any upstream dependency, anything not declared in the `Pipfile`, is free to be whatever version pip decides. – Eric Ihli May 07 '19 at 18:46
  • As far as I can tell, it's not that *his* Pipfile.lock doesn't match *his* environment, as you mention in your comment. It's that his `pip freeze` after a `pipenv install` doesn't match his boss's `pip freeze` after a `pipenv install`. And that is to be expected. – Eric Ihli May 07 '19 at 18:53
  • Just realized you were the one asking the question. I guess it really *doesn't* address the issue :) I'm not sure the difference between `pipenv lock` and `pip freeze` and why there would be differences. Curious to know. – Eric Ihli May 07 '19 at 19:09
  • Actually `pipenv install` installs from `Pipfile.lock` as well. The documentation is really inaccurate. – lapin Jun 07 '20 at 11:14