Custom Module Import Issue

Question

I can't seem to import my own custom NYT module. My project structure is as follows and I'm on a mac:

articulation/
    articulation/
        __init__.py # empty
        lib/
            nyt.py
            __init__.py # empty
        tests/
            test_nyt.py
            __init__.py # empty

When I try running python articulation/tests/test_nyt.py from that first parent directory, I get

  File "articulation/tests/test_nyt.py", line 5, in <module>
    from articulation.lib.nyt import NYT
    ImportError: No module named articulation.lib.nyt

I also tried

(venv) Ericas-MacBook-Pro:articulation edohring$ Python -m articulation/tests/test_nyt.py 
/Users/edohring/Desktop/articulation/venv/bin/Python: Import by filename is not supported.

test_nyt.py

import sys
sys.path.insert(0, '../../')
import unittest
#from mock import patch
# TODO: store example as fixture and complete test

from articulation.lib.nyt import NYT

class TestNYT(unittest.TestCase):
    @patch('articulation.lib.nyt.NYT.fetch')
    def test_nyt(self):
        print "hi"
        #assert issubclass(NYT, Article)
        # self.assertTrue(sour_surprise.title == '')"""


nyt.py

        from __future__ import division

import regex as re
import string
import urllib2
from collections import Counter

from bs4 import BeautifulSoup
from cookielib import CookieJar

PARSER_TYPE = 'html.parser'


class NYT:
    def __init__(self, title, url):
        self.url = url
        self.title = title
        self.words = get_words(url)


def get_words(url):
    cj = CookieJar()
    opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
    p = opener.open(url)
    soup = BeautifulSoup(p.read(), PARSER_TYPE)
    # title = soup.html.head.title.string
    letters = soup.find_all('p', class_='story-body-text story-content')
    if len(letters)==0:
        letters = soup.find_all('p', class_='paragraph--story')
    if len(letters)==0:
        letters = soup.find_all('p', class_='story-body-text', )
    words = Counter()
    for element in letters:
        a = element.get_text().split()
        for c in a:
            c = ''.join(ch for ch in c if c.isalpha())
            c = c.lower()
            if len(c) > 0: 
                words[c] += 1
    return words



def test_nyt():
    china_apple_stores = NYT('title_test', 'http://www.nytimes.com/2016/12/29/technology/iphone-china-apple-stores.html?_r=0')
    assert(len(china_apple_stores.words) > 0)
    # print china_apple_stores.words
    fri_brief = NYT('Russia, Syria, 2017: Your Friday Briefing', 'http://www.nytimes.com/2016/12/30/briefing/us-briefing-russia-syria-2017.html')
    assert(fri_brief.title == 'Russia, Syria, 2017: Your Friday Briefing')
    assert(fri_brief.url == 'http://www.nytimes.com/2016/12/30/briefing/us-briefing-russia-syria-2017.html')
    assert(len(fri_brief.words) > 0)
    vet = NYT('title_test', 'http://lens.blogs.nytimes.com/2017/01/03/a-love-story-and-twins-for-a-combat-veteran-amputee/')
    assert(len(vet.words)>0)
    print "All NYT Tests Passed"




#test_nyt()

I've tried the following and none seem to work - does anyone know how to fix this? - Adding an init.py file to the top directory -> Doesn't help
- Entering Memory Python couldn't find this - maybe because I'm using Python 2. If this is the issue I can post more what I tried. - Adding sys.path at the top from suggestion below

Answer 1

Doing this:

import sys
sys.path.insert(0, '../../')

is usually a bad idea. Sometimes it's useful for when you're testing something, or you have a single-use program that you just need to work for a short time and then you're going to throw away, but in general it's a bad habit to get into because it might stop working once you move directories around or once you give the code to someone else. I would advise you not to let yourself get in the habit of doing that.

The most likely reason to get the kind of error you're seeing is that the directory /Users/edohring/Desktop/articulation does not appear in sys.path. The first thing to do is see what actually is in sys.path, and one good way to do that is to temporarily put these lines at the top of test_nyt.py:

import os.path, sys
for p in sys.path:
    print(p)
    if not os.path.isabs(p):
        print('  (absolute: {})'.format(os.path.abspath(p)))
sys.exit()

Then run

python articulation/tests/test_nyt.py

and look at the output. You will get a line for each directory path that Python looks in to find its modules, and if any of those paths are relative, it will also print out the corresponding absolute path so that there is no confusion. I suspect you will find that /Users/edohring/Desktop/articulation does not appear anywhere in this list.

If that turns out to be the case, the most straightforward (but least future-proof) way to fix it is to run

export PYTHONPATH=".:$PYTHONPATH"

in the shell (not in Python!) before you use Python itself to do anything using your module. Directories named in the PYTHONPATH environment variable will be added to sys.path when Python starts up. This is only a temporary fix, unless you put it in a file like $HOME/.bashrc which will get read by the shell every time you open up a Terminal window. You can read about this and better ways to add the proper directory to sys.path in this question.

Perhaps a better way to run your script is to use the shell command

python -m articulation.tests.test_nyt

This needs to be run in the directory /Users/edohring/Desktop/articulation, or at least that directory needs to appear in sys.path in order for the command to work. But using the -m switch in this way causes Python to handle how it sets up sys.path a little differently, and it may work for you. You can read more about how sys.path is populated in this answer.