2

UPDATE Getting close. Now I'm running f2py on the .pyf file that should generate the _glmnet module.

I build the package python-glmnet packet with the following command. 

python setup.py config_fc --fcompiler=gnu95         
                                           --f77flags='-fdefault-real-8'         
                                          --f90flags='-fdefault-real-8' build

But when I import the module I get this error:

File "/Users/rose/221/tagger/tagger/glmnet/glmnet.py", line 2, in import _glmnet ImportError: No module named _glmnet

How can I import that module?

The glmnet directory also contains a glmnet.pyf file that begins with the following:

!    -*- f90 -*-
! Note: the context of this file is case sensitive.

python module _glmnet ! in
    interface  ! in :_glmnet
        subroutine elnet(ka,parm,no,ni,x,y,w,jd,vp,ne,nx,nlam,flmin,ulam,thr,isd,lmu,a0,ca,ia,nin,rsq,alm,nlp,jerr) ! in :glmnet:glmnet.f
            integer optional :: ka=1 ! Use covariance updates over naive by default
            real*8 :: parm
            integer intent(hide),check(shape(x,0)==no),depend(x) :: no=shape(x,0)
            integer intent(hide),check(shape(x,1)==ni),depend(x) :: ni=shape(x,1)
            real*8 dimension(no,ni) :: x
            real*8 dimension(no),depend(no) :: y
            real*8 dimension(no),depend(no) :: w
            integer dimension(*) :: jd
            real*8 dimension(ni),depend(ni) :: vp
            integer optional,depend(x) :: ne=min(shape(x,1), nx)
            integer :: nx
            integer optional,check((flmin < 1.0 || len(ulam)==nlam)),depend(flmin,ulam) :: nlam=len(ulam)
            real*8 :: flmin
            real*8 dimension(nlam) :: ulam
            real*8 :: thr
            integer optional :: isd=1 ! Standardize predictors by default

UPDATE

Where can I find this _glmnet module? Is it contained in the glmnet.pyf file, as shown below? I tried adding this glment folder to my PYTHONPATH, but that didn't change anything.

~/221/tagger/tagger/glmnet master ls
__init__.py                  example_lasso_elastic_net.py glmnet.pyc
__init__.pyc                 glmnet.f                     glmnet.pyf
elastic_net.py               glmnet.py
~/221/tagger/tagger/glmnet master head -10 glmnet.pyf
!    -*- f90 -*-
! Note: the context of this file is case sensitive.

python module _glmnet ! in
    interface  ! in :_glmnet
        subroutine elnet(ka,parm,no,ni,x,y,w,jd,vp,ne,nx,nlam,flmin,ulam,thr,isd,lmu,a0,ca,ia,nin,rsq,alm,nlp,jerr) ! in :glmnet:glmnet.f
            integer optional :: ka=1 ! Use covariance updates over naive by default
            real*8 :: parm
            integer intent(hide),check(shape(x,0)==no),depend(x) :: no=shape(x,0)
            integer intent(hide),check(shape(x,1)==ni),depend(x) :: ni=shape(x,1)
~/221/tagger/tagger/glmnet master echo $PYTHONPATH
/Users/rose/221/tagger/tagger/glmnet:
~/221/tagger/tagger/glmnet master cd ..
~/221/tagger/tagger master python main.py
Traceback (most recent call last):
  File "main.py", line 14, in <module>
    from glmnet import glmnet
  File "/Users/rose/221/tagger/tagger/glmnet/glmnet.py", line 2, in <module>
    import _glmnet
ImportError: No module named _glmnet
Rose Perrone
  • 61,572
  • 58
  • 208
  • 243
  • What's in your `PYTHONPATH` environment variable? – Dan Dec 10 '13 at 22:29
  • Did you try putting the setup command all on one line? The installation instructions for the package have backslashes to indicate continuation. – Dan Dec 10 '13 at 23:43
  • I think your python distro is weirdly configured. It might be worthwhile to install the [anaconda binary python/numpy/scipy distribution](http://continuum.io/downloads). – Dan Dec 11 '13 at 00:14
  • I forked the Stanford version of `glmnet-Python` and finished packaging it. Currently only supports Python3. If anyone needs to fit a large model with regularization in python and struggle with `statsmodels`, give this a try. https://github.com/hanfang/glmnet_python – Han Fang Mar 02 '17 at 17:26

2 Answers2

2

As far as I can tell from the source, it's looking for the _gmlnet module, which is defined in gmlnet.pyf. gmlnet.pyf is not a python module, it's a set of additional instructions for a program called f2py, and python will ignore the .pyf file. You need to compile the .pyf file along with a fortran file using f2py. Use a command like this:

f2py -c --fcompiler=gnu95 gmlnet.pyf gmlnet.f

Try installing f2py and then reinstalling the gmlnet package.

Dan
  • 12,157
  • 12
  • 50
  • 84
  • Thanks! I did that. Now when I run `f2py` on glmnet.pyf, I see `Wrote C/API module "_glmnet" to file "./_glmnetmodule.c"`, but I still can't `from glmnet import _glmnet`. – Rose Perrone Dec 11 '13 at 00:00
  • You want to run `f2py` on `gmlnet.pyf`. – Dan Dec 11 '13 at 00:01
  • Sorry, that was a typo. I did. – Rose Perrone Dec 11 '13 at 00:03
  • @Rose Perrone: The full command would be something like `f2py -c --fcompiler=gnu95 gmlnet.pyf gmlnet.f`. This should create `_gmlnet.so`. See [this forum discussion](http://numpy-discussion.10968.n7.nabble.com/f2py-how-to-use-pyf-files-td13039.html), which is where I got my information. – Dan Dec 11 '13 at 00:03
  • I get `ImportError: No module named scipy_distutils.fcompiler`. – Rose Perrone Dec 11 '13 at 00:07
  • Is that when you run `f2py` or `setup.py`? – Dan Dec 11 '13 at 00:13
  • let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/42902/discussion-between-dan-and-rose-perrone) – Dan Dec 11 '13 at 00:19
  • When I run `f2py` with the options you mentioned. – Rose Perrone Dec 11 '13 at 00:28
  • You have to install `scipy_distutils` and possibly other packages. See the [`f2py` documentation](http://cens.ioc.ee/projects/f2py2e/). – Dan Dec 11 '13 at 00:36
  • @Rose Perrone: It looks like you're using a really old version of f2py. Newer versions are part of numpy. What do you get when you run `which -a f2py`? – Dan Dec 11 '13 at 00:40
  • `f2py -v` gives `2.45.241_1926`, which is the latest version. – Rose Perrone Dec 11 '13 at 00:43
  • @Rose Perrone: I'm looking for how many versions of f2py are installed on your machine. – Dan Dec 11 '13 at 00:43
  • Ah, 2 are installed. er, no, `which -a f2py` prints `/usr/local/bin/f2py /usr/local/bin/f2py` – Rose Perrone Dec 11 '13 at 00:44
  • @Rose Perrone: Yeah, I think you had f2py installed before you did anything. It's part of numpy. The new version you installed must be pre-numpy integration. Try uninstalling it and using f2py again. – Dan Dec 11 '13 at 00:46
  • `which -a numpy` returns nothing, but `sudo pip install numpy --upgrade` says the requirement is already up to date. How can I install `f2py`? – Rose Perrone Dec 11 '13 at 00:58
  • @Rose Perrone: `which` returns executables in your path. Numpy is a library and won't show up with `which`. To see if you have numpy installed, go into the python terminal and type `import numpy`. If numpy is already installed, f2py is already installed. The problem is you have two versions installed, and one of them needs `scipy_distutils`. You need to *uninstall* whichever one you installed earlier. – Dan Dec 11 '13 at 01:36
  • I suspect that when you reinstalled `f2py` it overwrote the (newer) version that came with your python installation. You might need to uninstall and then reinstall numpy. – Dan Dec 11 '13 at 01:37
  • 1
    Awesome! It all works now! Can you explain how the generation of a shared object file enabled python to import `_glmnet`? – Rose Perrone Dec 11 '13 at 01:45
  • @Rose Perrone: When you import a module `eggs` in python, it looks in all of the directories in `PYTHONPATH` (as well as some other hardcoded directories) for `eggs.py`, `eggs.pyc`, `eggs.pyo`, `eggs.so` (unix only), `eggs.pyd` (windows only) or directories (in the case you imported `eggs.spam`, it would look for /whatever/eggs/spam.py). If it can't find what you wanted to import, it raises an exception. – Dan Dec 11 '13 at 01:55
  • `f2py` makes special pythony shared object files that can be loaded by python programs. Python is capable of running the code in `.so` files just like anything else, except I don't think it uses the linker daemon (LD_LIBRARY_PATH has no effect on python). – Dan Dec 11 '13 at 02:00
  • Recall that "object" in "shared object" refers to object code and not the object in "object-oriented". The `.so` file is annotated with the names of the binary functions inside, as well as some special stuff that tells python what modules things are in. In this case, if you read the `.pyf` file, you can see it's supposed to compile down to a module that contains `_gmlnet`. – Dan Dec 11 '13 at 02:05
0

Without knowing too much about what else is going on, this looks like python doesn't know where the _gmlnet package is located. If you haven't already, check the contents of your PYTHONPATH environment variable with echo $PYTHONPATH. If the _gmlnet directory isn't in there, you need to add it.

  • To add something to PYTHONPATH in bash, run the following command:

    export PYTHONPATH=/path/to/_gmlnet/directory:$PYTHONPATH

  • You can check that python has parsed this correctly by going into the interactive terminal and typing

    import sys

    print sys.path

  • Lastly, you likely don't want to have to type this into your shell each time you start up your terminal. For Mac OS, add the following to the end of your ~/.bash_profile file if you use bash. Make sure that this file ends with a newline.

    export PYTHONPATH=/path/to/_gmlnet/directory:$PYTHONPATH

Dan
  • 12,157
  • 12
  • 50
  • 84