From virtualenv, pip freeze > requirements.txt give TONES of garbage! How to trim it out?

Question

I'm following this tutorial: http://devcenter.heroku.com/articles/django

At some point I'm suposed to do:

pip freeze > requirements.txt

(Ofc. from virtualenv created instance of python)

And I get this:

(venv)przemoli@ubuntu:~/Programowanie/hellodjango$ cat requirements.txt 
BeautifulSoup==3.2.0
Brlapi==0.5.5
CherryPy==3.1.2
ClientForm==0.2.10
Django==1.3
GnuPGInterface==0.3.2
PAM==0.4.2
PIL==1.1.7
Routes==1.12.3
Twisted-Core==11.0.0
Twisted-Names==11.0.0
Twisted-Web==11.0.0
WebOb==1.0.8
adium-theme-ubuntu==0.3.1
apt-xapian-index==0.44
apturl==0.5.1ubuntu1
chardet==2.0.1
command-not-found==0.2.44
configglue==1.0
cssutils==0.9.8a1
defer==1.0.2
distribute==0.6.19
django-tagging==0.3.1
dnspython==1.9.4
duplicity==0.6.15
gnome-app-install==0.4.7-nmu1ubuntu2
httplib2==0.7.2
jockey==0.9.4
keyring==0.6.2
launchpadlib==1.9.8
lazr.restfulclient==0.11.2
lazr.uri==1.0.2
louis==2.3.0
lxml==2.3
mechanize==0.1.11
nvidia-common==0.0.0
oauth==1.0.1
onboard==0.96.1
oneconf==0.2.6.7
papyon==0.5.5
pexpect==2.3
piston-mini-client==0.6
protobuf==2.4.0a
psycopg2==2.4.4
pyOpenSSL==0.12
pycrypto==2.3
pycups==1.9.59
pycurl==7.19.0
pyinotify==0.9.1
pyparsing==1.5.2
pyserial==2.5
pysmbc==1.0.10
python-apt==0.8.0ubuntu9
python-dateutil==1.4.1
python-debian==0.1.20ubuntu2
python-virtkey==0.60.0
pyxdg==0.19
sessioninstaller==0.0.0
simplejson==2.1.6
system-service==0.1.6
ubuntu-sso-client==1.4.0
ubuntuone-couch==0.3.0
ubuntuone-installer==2.0.0
ubuntuone-storage-protocol==2.0.0
ufw==0.30.1-2ubuntu1
unattended-upgrades==0.1
usb-creator==0.2.23
virtualenv==1.6.4
wadllib==1.2.0
wsgiref==0.1.2
xdiagnose==1.1
xkit==0.0.0
zope.interface==3.6.1

When deploying on heroku it fails at Brlapi .....

I see lots of stuff from my main python installation which is on ubuntu. Which is BAD since Ubuntu use python for quite a few thing itself (ubuntu-one, usb-creator, etc..).

I do not need them on heroku! I need only Django, psycopg2, and their dependencies. I do not even know if its fault of pip, or virutalenv. (If you want to know my setup look at link above I copied it into terminal)

zsquare · Accepted Answer · 2015-12-17T20:04:33.743

That is one thing that has bugged me too quite a bit. This happens when you create a virtualenv without the --no-site-packages flag.

There are a couple of things you can do:

Create virtualenv with the --no-site-packages flag.
When installing apps, dont run pip install <name> directly, instead, add the library to your requirements.txt first, and then install the requirements. This is slower but makes sure your requirements are updated.
Manually delete libraries you dont need. A rule of thumb i follow for this is to add whatever is there in my INSTALLED_APPS, and database adapters. Most other required libraries will get installed automatically because of dependencies. I know its silly, but this is what I usually end up doing.

-- Edit --

I've since written a couple of scripts to help manage this. The first runs pip freeze and adds the found library to a provided requirements file, the other, runs pip install, and then adds it to the requirements file.

function pipa() {
    # Adds package to requirements file.
    # Usage: pipa <package> <path to requirements file>
    package_name=$1
    requirements_file=$2
    if [[ -z $requirements_file ]]
    then
        requirements_file='./requirements.txt'
    fi
    package_string=`pip freeze | grep -i $package_name`
    current_requirements=`cat $requirements_file`
    echo "$current_requirements\n$package_string" | LANG=C sort | uniq > $requirements_file
}

function pipia() {
    # Installs package and adds to requirements file.
    # Usage: pipia <package> <path to requirements file>
    package_name=$1
    requirements_file=$2
    if [[ -z $requirements_file ]]
    then
        requirements_file='./requirements.txt'
    fi
    pip install $package_name
    pipa $package_name $requirements_file
}

Note: since v. 1.7, virtualenv takes `--no-site-packages` by default, so it's not required you specify that option. It's the default. Link: http://pypi.python.org/pypi/virtualenv#changes-news — Paolo, Feb 25 '12 at 14:39
Thx! --no-site-packages works fine! (Though I'm still puzzled why heroku crew did not mentioned it in theirs docs). (And yes I did used Ubuntu 11.10 provided package containing virtualenv 1.6.4 :( :( :( ) — przemo_li, Feb 25 '12 at 20:36
If you're using the latest version of virtualenv, `--no-site-packages` isn't necessary anymore. I highly recommend not relying on python modules from aptitude :) — Kenneth Reitz, Apr 02 '12 at 21:13
You can also do `pip freeze --local > requirements.txt` This will output only the packages installed into your virtual env, without listing all dependencies (the packages themselves, handle those.) — Nick Zalutskiy, Mar 28 '13 at 13:31

score 25 · Answer 2 · answered Jan 17 '17 at 21:48

25

pipreqs solves the problem. It generates project-level requirement.txt file.

Install pipreqs: pip install pipreqs
Generate project-level requirement.txt file: pipreqs /path/to/your/project/ requirements file would be saved in /path/to/your/project/requirements.txt

answered Jan 17 '17 at 21:48

Haifeng Zhang

30,077
19
81
125

Sadly it does not work, it creates a requirements but still some modules are missing – Roman Feb 07 '17 at 16:17
1

@Roman if you use virtualenv to run your project,pls make sure you use pip that belongs to the virtualenv install `pipreqs` – Haifeng Zhang Feb 07 '17 at 16:32
Seems to be working as intended for me, would recommend as a quick fix. – SeedyROM May 26 '17 at 03:29
I can't seem to get it working. "No module named pipreqs.__main__; 'pipreqs' is a package and cannot be directly executed" – lmblanes Jul 20 '18 at 16:08
@lmblanes check whether your pip outdated – Haifeng Zhang Jul 20 '18 at 16:31
and if you are in the current directory use `pipreqs .` – Faizan Ahmad Jan 24 '22 at 08:18

score 10 · Answer 3 · answered Feb 25 '12 at 19:44

If you care a lot about the cleanliness of your requirements.txt you should not only use the --no-site-packages option as already mentioned but also consider not to pipe the output of pip freeze directly to your requirements.txt. The reason for that is, that when doing a pip freeze not only packages specified by yourself show up, but also dependencies installed by these packages! It isn't necessary to keep them all in your requirements.txt as they will get installed automatically with the package that requires them... So if you add a new package to your virtualenv you probably should just add the line for this package to your requirements.txt...

See this example:

(demo)[~]$ pip freeze
distribute==0.6.19
wsgiref==0.1.2
(demo)[~]$ pip install django-blog-zinnia
Downloading/unpacking django-blog-zinnia
  Downloading django-blog-zinnia-0.9.tar.gz (523Kb): 523Kb downloaded
  Running setup.py egg_info for package django-blog-zinnia

    no previously-included directories found matching 'docs/api'
    no previously-included directories found matching 'docs/build'
    no previously-included directories found matching 'docs/coverage'
    no previously-included directories found matching 'zinnia/media/zinnia/css/.sass-cache'
Downloading/unpacking BeautifulSoup>=3.2.0 (from django-blog-zinnia)
  Downloading BeautifulSoup-3.2.1.tar.gz
  Running setup.py egg_info for package BeautifulSoup

  # truncated as it installs some more dependencies
Successfully installed django-blog-zinnia BeautifulSoup django-mptt django-tagging django-xmlrpc pyparsing
Cleaning up...
(demo)[~]$ pip freeze
BeautifulSoup==3.2.1
distribute==0.6.19
django-blog-zinnia==0.9
django-mptt==0.5.2
django-tagging==0.3.1
django-xmlrpc==0.1.3
pyparsing==1.5.6
wsgiref==0.1.2

(Though I should probably mentioned that in most cases it will not hurt that you have these dependencies there, just your file will grow and get harder to maintain.)

Nah! I just wanted quick solution to separate my OS related python libraries from my django development libraries that should be installed on deployment machine! (imho default behaviour of pip freeze should be changed...) — przemo_li, Mar 14 '12 at 22:07
I think you absolutely should keep all the indirect dependencies in your requirements.txt actually, along with their version numbers. That way your builds are repeatable. I used to keep my direct requirements at the top of the file, and my indirect ("transitive") dependencies at the bottom. Now I just keep them all in one file, using pip freeze to generate the information, but with comments, and in alphabetical order. — tobych, Jan 29 '13 at 18:40

score 1 · Answer 4 · answered May 19 '21 at 19:40

1

You can use:

pip freeze --local > requirement.txt

so only the packages installed locally in your virtualenv are listed in requirements.txt, not the globally-installed packages.

answered May 19 '21 at 19:40

René Pijl

4,310
1
19
25

Mebatsion Sahle · Answer 5 · 2021-10-16T00:40:59.357

0

Simply,
pip3 freeze requirements.txt
then if you wanted to install all
pip3 install -r requirements.txt

edited Oct 16 '21 at 00:40

answered Oct 15 '21 at 08:26

Mebatsion Sahle

409
2
9

score -1 · Answer 6 · answered Apr 25 '22 at 08:31

-1

It is bad to use pip freeze to create the requirements file... You should manage your dependencies manually!

I've created a script to fix this issue (I've already been in dependency conflict hell).

answered Apr 25 '22 at 08:31

eyalyoli

1,677
17
16

From virtualenv, pip freeze > requirements.txt give TONES of garbage! How to trim it out?

6 Answers6

Linked

Related