What is the preferred way to reuse Python code across multiple projects

Question

I have ended up lot of small functions and code snippets which I use in of my projects.

For example, wrappers around libraries to do common tasks:

def DownloadFile(url, filename):
    output = open(filename,'wb')
    output.write(urllib.request.urlopen(url).read())
    output.close()

or, a quick way of trapping and ignoring errors:

def deleteFile(f):
    if f == "":
        pass
    else:
        try:
            os.remove(f)
        except:
            print("Cant delete ",f)

I have a lot of these functions, I am calling them using

sys.path.append('../../blah')
import lib_file as fle

My question is, what would be the preferred way of handling this :

a) put them into a proper library and import it

b) get proficient at using the standard Python libraries so you don't need wrappers

c) is there another way to resuse Python snippets? (apart from copy paste)

dilbert · Accepted Answer · 2014-05-01T05:09:39.073

I've dealt with this issue using revision control; in my case, Mercurial.

Lets say I have this directory structure:


Projects\python\
                general_libs\
                             .hg
                             __init__.py
                             acldict.py
                             easy_csv.py
                             easy_tar.py
                             fake_file.py
                             list_extract.py
                             state_machine.py
                             tag_extract.py
                             timeout.py
                             user_agents.py
                scraper_libs\
                             .hg
                             __init__.py
                             something_else.py
                py_prog\
                        libs\

I'm writing a program called py_prog. I wish to use code from the libraries general_libs and scraper_libs but I want the method to maintain 'linkage' between the 'copy' and the original location.

In this instance, both general_libs and scraper_libs are mercurial repositories.

Mercurial (and git) offer the clone command. This command clones the source repository to (if not otherwise specified) the current directory. With a cloned repository, you are able to push and pull updates to and from the original repository without needing to recopy or overwrite.

Navigating in a console to Projects\python\py_prog\libs I run the mercurial commands:

Projects\python\py_prog\libs> hg clone ..\..\general_libs
Projects\python\py_prog\libs> hg clone ..\..\scraper_libs

The directory structure now looks like:


Projects\python\
                general_libs\
                             .hg
                             __init__.py
                             acldict.py
                             easy_csv.py
                             easy_tar.py
                             fake_file.py
                             list_extract.py
                             state_machine.py
                             tag_extract.py
                             timeout.py
                             user_agents.py
                scraper_libs\
                             .hg
                             __init__.py
                             something_else.py
                py_prog\
                        py_prog.py
                        libs\
                             general_libs\
                                          .hg
                                          __init__.py
                                          acldict.py
                                          easy_csv.py
                                          easy_tar.py
                                          fake_file.py
                                          list_extract.py
                                          state_machine.py
                                          tag_extract.py
                                          timeout.py
                                          user_agents.py
                             scraper_libs\
                                          .hg
                                          __init__.py
                                          something_else.py

Now I can import these libraries functions without changing the system path.

py_prog.py:

from libs.scraper_libs.something_else import something

Alternatively, you could achieve a similar outcome (although without the other benefits of revision control) using symlinks (in *nix) or junction points (in Windows).

Do you script the clone, or just at the start of a project / when the library changes? — acutesoftware, May 01 '14 at 13:13
You typically administer the change behaviour manually. I believe you can `hg push` from the original libraries which will updates its clones; so that's a single operation. — dilbert, May 01 '14 at 22:41

What is the preferred way to reuse Python code across multiple projects

1 Answers1