How to store result of pip command into Pandas Datafarme

Question

For getting the list of installed libraries, I run the following command in Jupyter Notebook:

!pip list

and I get the following as output:

Package                            Version
---------------------------------- ---------
alabaster                          0.7.12
altair                             4.1.0
anaconda-client                    1.7.2
anaconda-navigator                 1.9.7
anaconda-project                   0.8.3
asn1crypto                         0.24.0
astroid                            2.2.5
astropy                            3.2.1
atomicwrites                       1.3.0
attrs                              20.3.0
alabaster                          0.7.12
altair                             4.1.0
anaconda-client                    1.7.2
anaconda-navigator                 1.9.7
anaconda-project                   0.8.3
asn1crypto                         0.24.0
astroid                            2.2.5
astropy                            3.2.1
atomicwrites                       1.3.0
attrs                              20.3.0
Automat                            20.2.0
azureml                            0.2.7
Babel                              2.7.0
backcall                           0.1.0
backports.functools-lru-cache      1.5
backports.os                       0.1.1
backports.shutil-get-terminal-size 1.0.0
backports.tempfile                 1.0
backports.weakref                  1.0.post1
beautifulsoup4                     4.7.1
bitarray                           0.9.3
bkcharts                           0.2
bleach                             3.1.0
bokeh                              1.2.0
boto                               2.49.0
boto3                              1.14.61
botocore                           1.17.61
Bottleneck                         1.2.1
branca                             0.4.1
category-encoders                  2.2.2
certifi                            2019.6.16
cffi                               1.12.3
chardet                            3.0.4
Click                              7.0
click-spinner                      0.1.10
cloudpickle                        1.2.1
clyent                             1.2.2
colorama                           0.4.1
colorlog                           4.2.1
comtypes                           1.1.7
conda                              4.7.10
conda-build                        3.18.8
conda-package-handling             1.3.11
conda-verify                       3.4.2
configparser                       5.0.1
constantly                         15.1.0
contextlib2                        0.5.5
coverage                           5.3
coveralls                          2.1.2
cryptography                       2.7
cycler                             0.10.0
Cython                             0.29.14
cytoolz                            0.10.0
dacite                             1.5.1
dask                               2.1.0
decorator                          4.4.0
defusedxml                         0.6.0
distributed                        2.1.0
docopt                             0.6.2
docutils                           0.14
entrypoints                        0.3
et-xmlfile                         1.0.1
fastcache                          1.1.0
feature-engine                     0.6.1
filelock                           3.0.12
Flask                              1.1.1
flit-core                          2.3.0
folium                             0.11.0
furl                               2.1.0
future                             0.17.1
gensim                             3.8.3
genson                             1.2.2
gevent                             1.4.0
glob2                              0.7
greenlet                           0.4.15
h5py                               2.9.0
heapdict                           1.0.0
html5lib                           1.0.1
hyperlink                          20.0.1
hypothesis                         5.41.2
idna                               2.8
imageio                            2.5.0
imagesize                          1.1.0
imbalanced-learn                   0.7.0
importlib-metadata                 0.17
incremental                        17.5.0
ipykernel                          5.1.1
ipython                            7.6.1
ipython-genutils                   0.2.0
ipywidgets                         7.5.0
isort                              4.3.21
itsdangerous                       1.1.0
jdcal                              1.4.1
jedi                               0.13.3
Jinja2                             2.11.2
jmespath                           0.10.0
joblib                             0.13.2
json5                              0.8.4
jsonschema                         3.2.0
jupyter                            1.0.0
jupyter-client                     5.3.1
jupyter-console                    6.0.0
jupyter-core                       4.5.0
jupyterlab                         1.0.2
jupyterlab-server                  1.0.0
keyring                            18.0.0
kiwisolver                         1.1.0
lazy-object-proxy                  1.4.1
libarchive-c                       2.8
llvmlite                           0.29.0
locket                             0.2.0
lxml                               4.5.2
MarkupSafe                         1.1.1
matplotlib                         3.1.0
mccabe                             0.6.1
menuinst                           1.4.16
mistune                            0.8.4
mkl-fft                            1.0.12
mkl-random                         1.0.2
mkl-service                        2.0.2
mock                               3.0.5
more-itertools                     7.0.0
mpmath                             1.1.0
msgpack                            0.6.1
multipledispatch                   0.6.0
munch                              2.5.0
navigator-updater                  0.2.1
nbconvert                          5.6.1
nbformat                           4.4.0
networkx                           2.3
nibabel                            3.1.1
nltk                               3.4.4
nose                               1.3.7
notebook                           6.0.0
numba                              0.44.1
numexpr                            2.6.9
numpy                              1.19.2
numpydoc                           0.9.1
olefile                            0.46
openpyxl                           2.6.2
orderedmultidict                   1.0.1
packaging                          20.4
pandas                             1.1.3
pandocfilters                      1.4.2
parso                              0.5.0
partd                              1.0.0
path.py                            12.0.1
pathlib2                           2.3.4
patsy                              0.5.1
pep517                             0.8.2
pep8                               1.7.1
pickleshare                        0.7.5
Pillow                             6.1.0
pip                                20.2.4
pkginfo                            1.5.0.1
plotly                             4.9.0
pluggy                             0.12.0
ply                                3.11
pmdarima                           1.7.1
prometheus-client                  0.7.1
prompt-toolkit                     2.0.9
psutil                             5.6.3
psycopg2                           2.8.6
py                                 1.8.0
pycodestyle                        2.5.0
pycosat                            0.6.3
pycparser                          2.19
pycrypto                           2.6.1
pycurl                             7.43.0.3
pyflakes                           2.1.1
Pygments                           2.4.2
PyHamcrest                         2.0.2
pylint                             2.3.1
pyodbc                             4.0.26
pyOpenSSL                          19.0.0
pyparsing                          2.4.0
pyreadline                         2.1
pyrsistent                         0.14.11
PySocks                            1.7.0
pytest                             5.0.1
pytest-arraydiff                   0.3
pytest-astropy                     0.5.0
pytest-cov                         2.10.1
pytest-doctestplus                 0.3.0
pytest-openfiles                   0.3.2
pytest-remotedata                  0.3.1
python-dateutil                    2.8.0
pytoml                             0.1.21
pytz                               2019.1
PyWavelets                         1.0.3
pywin32                            223
pywinpty                           0.5.5
PyYAML                             5.3.1
pyzmq                              18.0.0
QtAwesome                          0.5.7
qtconsole                          4.5.1
QtPy                               1.8.0
requests                           2.22.0
requests-toolbelt                  0.9.1
retrying                           1.3.3
rope                               0.14.0
ruamel-yaml                        0.15.46
ruamel.yaml                        0.16.12
ruamel.yaml.clib                   0.2.2
s3transfer                         0.3.3
scikit-image                       0.15.0
scikit-learn                       0.23.2
scipy                              1.5.3
seaborn                            0.10.1
Send2Trash                         1.5.0
setuptools                         41.0.1
simplegeneric                      0.8.1
simplejson                         3.17.2
singledispatch                     3.4.0.3
six                                1.12.0
sklearn                            0.0
smart-open                         2.1.1
snowballstemmer                    1.9.0
sortedcollections                  1.1.2
sortedcontainers                   2.1.0
soupsieve                          1.8
Sphinx                             2.1.2
sphinxcontrib-applehelp            1.0.1
sphinxcontrib-devhelp              1.0.1
sphinxcontrib-htmlhelp             1.0.2
sphinxcontrib-jsmath               1.0.1
sphinxcontrib-qthelp               1.0.2
sphinxcontrib-serializinghtml      1.1.3
sphinxcontrib-websupport           1.1.2
spyder                             3.3.6
spyder-kernels                     0.5.1
SQLAlchemy                         1.3.5
statsmodels                        0.11.1
stumpy                             1.5.0
sympy                              1.4
tables                             3.5.2
tabpy                              2.3.1
tabulate                           0.8.7
tblib                              1.4.0
terminado                          0.8.2
testpath                           0.4.2
textblob                           0.15.3
threadpoolctl                      2.1.0
toml                               0.10.1
toolz                              0.10.0
tornado                            6.0.3
tqdm                               4.32.1
traitlets                          4.3.2
Twisted                            20.3.0
unicodecsv                         0.14.1
urllib3                            1.24.2
validators                         0.16.0
wcwidth                            0.1.7
webencodings                       0.5.1
Werkzeug                           0.15.4
wheel                              0.33.4
widgetsnbextension                 3.5.0
win-inet-pton                      1.1.0
win-unicode-console                0.5
wincertstore                       0.2
wrapt                              1.11.2
xlrd                               1.2.0
XlsxWriter                         1.1.8
xlwings                            0.15.8
xlwt                               1.3.0
zict                               1.0.0
zipp                               0.5.1
zope.interface                     5.2.0

How to store this result into a Pandas Dataframe? Just copying the data and pasting in two lists would do the job, but is there any code-based alternative to achieve this? In short, how to store the result of pip command into a Pandas Dataframe?

P.S: I referred here but was not of any help.

Answer 1

We can use os module to create the pip list, then we use pandas.read_csv with \s+ as seperator to read the pip list into a dataframe:

import os
import pandas as pd

# create pip list txt
os.system('pip list > pip_list.txt')

# read content into pandas df
df = pd.read_csv('pip_list.txt', sep=r'\s+', skiprows=[1])

# clean up
os.remove('pip_list.txt')

             Package  Version
0            appnope    0.1.0
1        argon2-cffi   20.1.0
2            astroid    2.4.2
3    async-generator     1.10
4              attrs   20.2.0
..               ...      ...
100          urllib3  1.25.10
101          wcwidth    0.2.5
102     webencodings    0.5.1
103            wrapt   1.12.1
104             xlrd    1.2.0

[105 rows x 2 columns]

Answer 2

You can use subprocess to get the result in memory instead of creating a new file.

import subprocess
from io import StringIO
command = "pip list"
process = subprocess.Popen(command.split(), stdout=subprocess.PIPE)
output, error = process.communicate()
pd.read_csv(StringIO(output.decode("utf-8")),  sep=r"\s*", skiprows=[1])