2

So, I have this project, where I use yapsy and watchdog, but both those libraries have broken versions on PyPI (at least for p3, what I call "broken" was just ported the wrong way).

Instead of installing them from PyPI, I cloned their GIT repositories and installed them by hand. Problem came up when I started writing setup.py.

How do I tell installing app (pip, easy_install, whatever) to use version from VCS repo, instead of PyPI?

I could do this by forking and fixing those libs, but I find it... wrong. I would have to freeze library, or wait some time, until someone merges my fix to libs repository.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
Filip Malczak
  • 3,124
  • 24
  • 44
  • Closely related: [Additional actions in setup.py for install](http://programmers.stackexchange.com/q/223263) – Martijn Pieters Jan 07 '14 at 13:02
  • In summary: if you dictate in your `setup.py` how dependencies are installed, you tie yourself too closely to those dependencies. You'll need to upgrade your own version number every time one how one of the dependencies switches download locations. Provide a `requirements.txt` or `buildout.cfg` or manual instructions in your project documentation instead. – Martijn Pieters Jan 07 '14 at 13:03
  • See https://caremad.io/blog/setup-vs-requirement/ for a good blog post on the subject. – Martijn Pieters Jan 07 '14 at 13:11

1 Answers1

1

Before you use this solution

I strongly recommend reading comments both to question and answer - this solution is dirty and should be used only, when there is no other choice (because of company policy, or anything else that you cannot change).

Solution

First thing I found out was that I need to use dependency link. It is useful, when repo version is higher than PyPI. Problem is, that installation apps prefer PyPI over VCS, when versions are the same.

So, thanks to this: Setuptools unable to use link from dependency_links I figured out that I need to tell installing app that VCS holds higher version than PyPI, even though they are the same, and require version less or equal to this higher version (which I declared is on VCS).

So, yeah, cool. I can write:

...
install_requires=[ ..., "watchdog<0.6.1", ...],
...
dependency_links = [
    ...
    "git+https://github.com/gorakhargosh/watchdog.git#egg=watchdog",
    ...
], ...

but if tommorow new version comes out, then I stay behind newest bug fixes, etc.

So I figured out, that I need to find out highest version at the moment, and do the trick with version "one point higher on last position". Here is code I used for this. I put it in "setup_helpers.py": http://pastebin.com/1crW5VCL

Now, in setup.py I did something like that:

from setup_helpers import vcs_requirement, egg_name
...
install_requires=[ ..., vcs_requirement("watchdog"), ...],
...
dependency_links = [
    ...
    "git+https://github.com/gorakhargosh/watchdog.git#egg=%s" % egg_name("watchdog"),
    ...
], ...

And that does the trick - and will as long, as noone will mess with version numbers (so they stay strictly numeric, without branches, etc, etc). Also, this enforces assumption that VCS code is more up-to-date than PyPI code. It works for me.

Community
  • 1
  • 1
Filip Malczak
  • 3,124
  • 24
  • 44
  • Your package **really** should not make such decisions. Package distributions can easily change over time, don't tie your package to a specific location. – Martijn Pieters Jan 07 '14 at 12:50
  • Depend on version ranges, and leave finding the packages up to `pip` or `buildout` or whatever tool you are using to install your package. They can be instructed to look at GitHub for a dependency. – Martijn Pieters Jan 07 '14 at 12:51
  • So, what do you propose in this situation? I understand WHY PyPI was created, and that repo may be moved, etc - but this is the case in which PyPI wasn't used properly, and I have no other choice, than install this from VCS repo. Of course, if fix will come up, I'll change my setup.py to use PyPI again. – Filip Malczak Jan 07 '14 at 12:52
  • If you need to change your setup.py just because an external dependency changed, you also need to change your version number, otherwise your package won't be reinstalled properly. Don't mess with `setup.py` unless your **own** code change.d – Martijn Pieters Jan 07 '14 at 12:55
  • See http://codeinthehole.com/writing/using-pip-and-requirementstxt-to-install-from-the-head-of-a-github-branch/ for a pip example. – Martijn Pieters Jan 07 '14 at 12:55
  • Seen that already. Do you mean that instead of specifying that in setup() arguments, I should also provide requirements.txt file together with setup.py? – Filip Malczak Jan 07 '14 at 12:57
  • Provide a `requirements.txt` *separately*, outside of your project packaging. You can include it in your source repository, just avoid that you have to release a new package every time one of your dependencies has to be installed differently. – Martijn Pieters Jan 07 '14 at 12:59
  • In other words: how dependencies are installed shouldn't be your problem. You can help your end-users figure that out, but in the end, it is the dependent project that should fix this, really. – Martijn Pieters Jan 07 '14 at 13:00
  • Closely related: [Additional actions in setup.py for install](http://programmers.stackexchange.com/q/223263) – Martijn Pieters Jan 07 '14 at 13:00
  • Thing is, place where I work has policy like "our programs should be installed like 'pip install project_name'". I'd like it to stay 'location-agnostic', but it's not my choice. Unless, there is some workaround for my situation, that will end in simple oneliner, as my boss wants? – Filip Malczak Jan 07 '14 at 13:03
  • And, yes - it is related, that is my post too :P – Filip Malczak Jan 07 '14 at 13:03
  • Ah, should I add more explanation as to why you should not do this there? – Martijn Pieters Jan 07 '14 at 13:04
  • Perhaps it is time to educate your boss instead? The one-liner would be `pip install -r requirements.txt`. – Martijn Pieters Jan 07 '14 at 13:06
  • I did read your comment. I did respond with: *change his mind*. Have him read this: https://caremad.io/blog/setup-vs-requirement/ – Martijn Pieters Jan 07 '14 at 13:09
  • I'm already trying to do so, but until I succeed, I have to do this the wrong way. Personally I agree with you, but it doesn't matter. What matters is that my solution is an effect of firm policy, and it's the only way to do things here for a moment. MAYBE someone will find this useful, if they are in similiar situation. For clarity, I will edit my answer and tell readers to read this discussion too. – Filip Malczak Jan 07 '14 at 13:11
  • Bottom line is: Don't make your package a slave of its dependencies when it comes to installing. If your package is only used internally, so be it, that's on your own head, but if you need to distribute this package to anyone else, then you just made their installation story hell too. If this is a package on PyPI I had to install, I'd move on to a different one or fork it. – Martijn Pieters Jan 07 '14 at 13:11