Imported code cannot open files in its directory

Question

I have the following file structure:

test/
    test1.py
test2.py
text.txt

Here are the contents of the files

test1.py:

import sys
sys.path.append('../')
import test2

test2.read()

test2.py:

def read():
    with open('text.txt', 'rb') as f:
        print f.read()

if __name__ == "__main__":
    read()

text.txt contains a line of text. When I run test1.py, I get a "File not found" error:

Traceback (most recent call last):
  File "test1.py", line 5, in <module>
    test2.read()
  File "../test2.py", line 2, in read
    with open('text.txt', 'rb') as f:
IOError: [Errno 2] No such file or directory: 'text.txt'

I kind of understand why this error is coming up. But how do I deal with these kind of errors. I would like the code in test2.py to be like library code which I can use anywhere.

This isn't actually the right path to add to `sys.path` in the first place. This adds the parent directory of the [current working directory](http://en.wikipedia.org/wiki/Working_directory). What you want to add is the parent of the directory _that `test1.py` is in_. Those _could_ be the same thing, but there's no guarantee they will be. For example, if you're sitting in the parent directory in the shell, you can always run `python test/test1.py`, and then everything will break. — abarnert, Jan 16 '14 at 00:21
@abarnert excuse my ignorance, but how can we add the parent of test1.py's directory? — Phani, Jan 16 '14 at 00:39
The safest way to do it (in 2.7) is: `os.path.dirname(os.path.abspath(__file__))`. (The `abspath` in there in case you `chdir` later on. The key part of it is starting from wherever `__file__` is—that is, the `test1.py` script—instead of wherever the working directory is.) — abarnert, Jan 16 '14 at 00:45

Elisha · Answer 1 · 2014-01-16T00:48:04.703

5

sys.path used for python path (PYTHONPATH eviroment variable). i.e. where to look for python libraries when you import some library. it dose not effect where open() is looking for files.

when you open(filename). the filename is relative to the procees working directory. (the path the code was run from)

so if you want to access a flie that its path is relative to the path of the code file, then you can use the builtin variable __file__ which hold the current file path.

so you can change test2.py to be:

import os

def read():
    with open(os.path.join(os.path.dirname(__file__),'text.txt'), 'rb') as f:
        print f.read()

edited Jan 16 '14 at 00:48

answered Jan 16 '14 at 00:05

Elisha

4,811
4
30
46

But, I will still need to use `sys.path.append()` for importing `test2.py` right? – Phani Jan 16 '14 at 00:14
1

Note that, although this is definitely the "right" way to do it, it won't work if you ever `chdir` while your app is running, and there are various edge cases where it doesn't actually work right (e.g., `__file__` ends up as `test2.py`—that's fixed in 3.4, where `__file__` is guaranteed to be an absolute path no matter what, but that doesn't help for 2.7). – abarnert Jan 16 '14 at 00:16
2

Also, I think mentioning the concept of ["current working directory"](http://en.wikipedia.org/wiki/Working_directory) with that Wikipedia link or some better link might make this clearer than just "where you run the code from". Because as written, it implies that the directory that `test1.py` lives in is what matters, and that isn't true. – abarnert Jan 16 '14 at 00:17
@abarnert I would argue that my solution actually the "right" way of doing it. – CrazyCasta Jan 16 '14 at 00:17
@Phani right, if `test2.py` path is not already in your `PYTHONPATH` then you still need to `sys.path.append()` it – Elisha Jan 16 '14 at 00:18
@CrazyCasta: Yes, if you can rely on your script being installed rather than run out of source (and are willing to either require `setuptools` or do the bootstrap thing, but I can't see why you wouldn't be willing to bootstrap it), `pkg_resources` is the best answer. – abarnert Jan 16 '14 at 00:19
@CrazyCasta, the problem with your suggestion is that it will break on different installation or when you copy your program to another location. – Elisha Jan 16 '14 at 00:20
@Elisha: The whole point of using `setuptools` and installing (to site-packages, or to a binary distribution of some kind, or whatever) is to make that problem irrelevant. The top-level script(s) end up on your PATH, and refer to your modules, and can be moved around freely; those modules end up in your `site-packages` or similar and never get moved around. Moving to a different installation is done by installing the same package on the new installation. – abarnert Jan 16 '14 at 00:23
@abarnert @CrazyCasta My code is more like ongoing research work which will not come to a "packaged" state any time soon. So, most of the time I will be running out of source. Can I use `pkg_resources` in this case? – Phani Jan 16 '14 at 00:26
2

@Phani: Even for ongoing research work, making it into an installable package makes all kinds of things easier for you. It means you can use `setuptools`. It means you can have 28 forked or branched versions of the source that all share the same data files. You can add a Cython or C extension trivially, while doing it manually without `setup.py` is a nightmare. You can document and automate your dependencies, so moving to a new machine or upgrading to a new Python is easy. And so on. – abarnert Jan 16 '14 at 00:28
@Phani AFAIK you can use pkg_resources even if you don't plan on installing it. I may be wrong as I'm not all that familiar with it, but that's my understanding. – CrazyCasta Jan 16 '14 at 00:31
My comments here about @CrazyCasta answer are irrelevant. I mixed up with his answer and @ alKid's by mistake – Elisha Jan 16 '14 at 06:20
@Elisha Lol, now it all makes sense at least :P – CrazyCasta Jan 16 '14 at 06:43

score 3 · Accepted Answer · edited May 23 '17 at 12:29

3

The proper way of doing what you're asking for is to use pkg_resources as described here. Basically something like the following would be what you want in test2.py:

import pkg_resources

def read():
    with pkg_resources.resource_stream(__name__, 'text.txt') as f:
        print f.read()

if __name__ == "__main__":
    read()

edited May 23 '17 at 12:29

Community

1
1

answered Jan 16 '14 at 00:07

CrazyCasta

26,917
4
45
72

Assuming you want to build this as a package to be installed (or built into a binary via `cx_Freeze`, or wrapped up in a deb/rpm, or…), this is the right answer. And usually, for anything more complicated than a single script, you _do_ want that. Being able to run out of the source tree is, at best, a "nice to have"; being able to run when installed is critical. – abarnert Jan 16 '14 at 00:26
@abarnert I'm not sure what you guys are criticizing. This does not require that the package be installed. I just tried this by creating the test2.py file inside a package called test2, running python, doing import test2.test2; test2.test2.read() and it worked for me. Are you saying that doesn't work for some people? – CrazyCasta Jan 16 '14 at 00:29

aIKid · Answer 3 · 2014-01-16T01:31:06.513

0

Don't use relative path, then. Use the full path:

with open(r'C:\Somewhere\someplace\test.txt') as f:

edited Jan 16 '14 at 01:31

answered Jan 16 '14 at 00:03

aIKid

26,968
4
39
65

That is not the correct full path. `..` means "go up a directory". – user2357112 Jan 16 '14 at 00:04
I want this to be part of a project that I run from multiple places, is there a cleaner solution? – Phani Jan 16 '14 at 00:04
Hm.. The code is _imported_ to `test1` script. So the one looking for the script is `test1`, not `test2`. I'll update my answer in a sec. – aIKid Jan 16 '14 at 00:06
This isn't likely to work well for "library code" if he wants to redistribute it. – CrazyCasta Jan 16 '14 at 00:12
There's also the fact that this path has a tab character in it, not a backslash and a `t`… – abarnert Jan 16 '14 at 00:24

Imported code cannot open files in its directory

3 Answers3