2

I'm trying to optimize our poetry build times by storing a tar.gz of the venv on the end of the build in an azure storage blob (an S3) with key generated by doing a md5sum of the poetry.lock and the pyproject.toml. (this works fine)

So, in the beginning of another build, I would hash those files and try to find if that blob exists in storage. If yes, download it and extract its contents to the .venv/ dir of the project.

This worked fine at the beginning but after a while I started getting builds that were throwing this error when trying to run a command through poetry: (any command)

+ poetry run reorder-python-imports --diff-only <SOME_FILES>

  FileNotFoundError

  [Errno 2] No such file or directory

  at /usr/local/lib/python3.8/os.py:591 in _execvpe
       587│         argrest = (args,)
       588│         env = environ
       589│ 
       590│     if path.dirname(file):
    →  591│         exec_func(file, *argrest)
       592│         return
       593│     saved_exc = None
       594│     path_list = get_exec_path(env)
       595│     if name != 'nt':

I have confirmed that there is no .venv folder already in the project dir and that the contents are exactly the same as if it were going to install it from the network.

If I just run poetry install on top of the cached venv, it says it has no more dependencies to install, but it throws the error above. If I delete the venv and install everything again, commands work fine.

I have no more ideas on how to debug and solve this issue. Help would be very appreciated! :)

Johny Serpa
  • 155
  • 1
  • 2
  • 16

1 Answers1

1

Ok, I think I've understood that virtualenvs have some hardcoded paths that are not easy to move around.

For what it says in this post Can I move a virtualenv? we should not move venvs as a good practice.

And there it goes my way of speeding up builds...

Johny Serpa
  • 155
  • 1
  • 2
  • 16
  • Hello, I am facing the same problem. You said "we should not move venvs as a good practice", but are you really moving them? Aren't you just zipping and unzipping to the exact same place? – Pedro A Sep 22 '22 at 17:17
  • In my case I noticed that after recreating it from scratch, I got different values for `version_info` and `virtualenv` in `.venv/pyenv.cfg`. This is probably because I tried to reuse the cache after upgrading python from 3.9.13 to 3.9.14. – Pedro A Sep 22 '22 at 17:27