Is pandas_ml broken?

Question

The version info and issue are as given below. I want to know if pandas_ml is broken or am I doing something wrong. Why am I not able to import pandas_ml?

Basic info: Versions of sklearn and pandas_ml and python are given below:

Python                            3.8.2
scikit-learn                      0.23.0
pandas-ml                         0.6.1

Issue:

import pandas_ml as pdml

returns the following error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-47-79d5f9d2381c> in <module>
----> 1 import pandas_ml as pdml
      2 #from pandas_ml import ModelFrame
      3 #mf = pdml.ModelFrame(df.to_dict())
      4 #mf.head()

d:\program files\python38\lib\site-packages\pandas_ml\__init__.py in <module>
      1 #!/usr/bin/env python
      2 
----> 3 from pandas_ml.core import ModelFrame, ModelSeries       # noqa
      4 from pandas_ml.tools import info                         # noqa
      5 from pandas_ml.version import version as __version__     # noqa

d:\program files\python38\lib\site-packages\pandas_ml\core\__init__.py in <module>
      1 #!/usr/bin/env python
      2 
----> 3 from pandas_ml.core.frame import ModelFrame       # noqa
      4 from pandas_ml.core.series import ModelSeries     # noqa

d:\program files\python38\lib\site-packages\pandas_ml\core\frame.py in <module>
      8 
      9 import pandas_ml.imbaccessors as imbaccessors
---> 10 import pandas_ml.skaccessors as skaccessors
     11 import pandas_ml.smaccessors as smaccessors
     12 import pandas_ml.snsaccessors as snsaccessors

d:\program files\python38\lib\site-packages\pandas_ml\skaccessors\__init__.py in <module>
     13 from pandas_ml.skaccessors.linear_model import LinearModelMethods                 # noqa
     14 from pandas_ml.skaccessors.manifold import ManifoldMethods                        # noqa
---> 15 from pandas_ml.skaccessors.metrics import MetricsMethods                          # noqa
     16 from pandas_ml.skaccessors.model_selection import ModelSelectionMethods           # noqa
     17 from pandas_ml.skaccessors.neighbors import NeighborsMethods                      # noqa

d:\program files\python38\lib\site-packages\pandas_ml\skaccessors\metrics.py in <module>
    254 _true_pred_methods = (_classification_methods + _regression_methods
    255                       + _cluster_methods)
--> 256 _attach_methods(MetricsMethods, _wrap_target_pred_func, _true_pred_methods)
    257 
    258 

d:\program files\python38\lib\site-packages\pandas_ml\core\accessor.py in _attach_methods(cls, wrap_func, methods)
     91 
     92         for method in methods:
---> 93             _f = getattr(module, method)
     94             if hasattr(cls, method):
     95                 raise ValueError("{0} already has '{1}' method".format(cls, method))

AttributeError: module 'sklearn.metrics' has no attribute 'jaccard_similarity_score'

Answer 1

It seems it is indeed. Here is the situation:

Although the function jaccard_similarity_score is not shown in the available ones of sklearn.metrics in the documentation, it was still there under the hood (hence available) until v0.22.2 (source code) in addition to the jaccard_score one. But in the source code of the latest v0.23, it has been removed, and only jaccard_score remains.

This would imply that it could still be possible to use pandas-ml by simply downgrading scikit-learn to v.0.22.2. But unfortunately this will not work either, throwing a different error:

!pip install pandas-ml
# Successfully installed enum34-1.1.10 pandas-ml-0.6.1

import sklearn
sklearn.__version__
# '0.22.2.post1'

import pandas_ml as pdml

[...]

AttributeError: module 'sklearn.preprocessing' has no attribute 'Imputer'

I guess it would be possible to find a scikit-learn version that works with it by going back enough (the last commit in their Github repo was in March 2019), but not sure if it is worth the fuss. In any case, they do not even mention scikit-learn (let alone any specific version of it) in their requirements file, which does not seem as sound practice, and the whole project seems rather abandoned.

Answer 2

So after some time and effort on this, I got it working and realized that the concept of broken in Python is rather murky. It would depend upon the combination of libraries you are trying to use and their dependencies. The older releases are all available and can be used but sometimes, it can be a hit-and-trial process to find that correct combination of package versions which gets everything working.

The other thing that I learnt from this exercise is the importance of having a significant expertise in creating and managing the virtual environments when programming with python.

In my case, I got help from some friends with the hit-and-trial part and found that pandas_ml works on python 3.7. Given below is the pip freeze output which can be used to setup a reliable virtual environment for machine learning and deep learning work using libraries like pandas_ml and imbalanced-learn libraries and may include some other libraries which have not had a new release in the last few years.

To create a working environment with the right version of packages which would ensure that pandas_ml and imbalanced-learn libraries work, create an environment with the following configuration on Python 3.7.

backcall==0.1.0
colorama==0.4.3
cycler==0.10.0
decorator==4.4.2
enum34==1.1.10
imbalanced-learn==0.4.3
ipykernel==5.2.1
ipython==7.14.0
ipython-genutils==0.2.0
jedi==0.17.0
joblib==0.15.0
jupyter-client==6.1.3
jupyter-core==4.6.3
kiwisolver==1.2.0
matplotlib==3.2.1
numpy==1.15.4
pandas==0.24.2
pandas-ml==0.6.1
parso==0.7.0
pickleshare==0.7.5
prompt-toolkit==3.0.5
Pygments==2.6.1
pyparsing==2.4.7
python-dateutil==2.8.1
pytz==2020.1
pywin32==227
pyzmq==19.0.1
scikit-learn==0.20.0
scipy==1.3.3
six==1.14.0
threadpoolctl==2.0.0
tornado==6.0.4
traitlets==4.3.3
wcwidth==0.1.9

Hope this helps someone who is looking for the right combination of library versions to setup their machine and deep learning environment in python using pandas_ml and imbalanced-learn packages.