412

The below code will not join, when debugged the command does not store the whole path but just the last entry.

os.path.join('/home/build/test/sandboxes/', todaystr, '/new_sandbox/')

When I test this it only stores the /new_sandbox/ part of the code.

Remi Guan
  • 21,506
  • 17
  • 64
  • 87
chrisg
  • 40,337
  • 38
  • 86
  • 107

16 Answers16

550

The latter strings shouldn't start with a slash. If they start with a slash, then they're considered an "absolute path" and everything before them is discarded.

Quoting the Python docs for os.path.join:

If a component is an absolute path, all previous components are thrown away and joining continues from the absolute path component.

Note on Windows, the behaviour in relation to drive letters, which seems to have changed compared to earlier Python versions:

On Windows, the drive letter is not reset when an absolute path component (e.g., r'\foo') is encountered. If a component contains a drive letter, all previous components are thrown away and the drive letter is reset. Note that since there is a current directory for each drive, os.path.join("c:", "foo") represents a path relative to the current directory on drive C: (c:foo), not c:\foo.

Craig McQueen
  • 41,871
  • 30
  • 130
  • 181
  • 108
    -1: **No** string should include a "/". One whole point of os.path.join is to prevent putting any slashes in the path. – S.Lott Dec 22 '09 at 12:29
  • 2
    this is not cool. if i explicitly ask to join those paths (or strings) do it if there is a leading '/' or not on that path/string. – aschmid00 Dec 16 '11 at 20:06
  • 2
    if you want that, just use str.join(), this is os.path and therefore it's consistent with its package semantics. – marcorossi Dec 16 '11 at 23:00
  • 11
    The problem with str.join() is, of course, that it won't eliminate double slashes. I think this is the primary purpose for folks using os.path.join. e.g. '/'.join(['/etc/', '/conf']) results in three slashes: '/etc///conf' – Dustin Rasener Jul 31 '12 at 14:03
  • 20
    @DustinRasener You can use [``os.path.normpath``](http://docs.python.org/2/library/os.path.html#os.path.normpath) to achieve that aim. – Gareth Latty Oct 28 '12 at 17:48
  • @S.Lott Are you saying the first string shouldn't start with a slash or that it shouldn't include a trailing slash? – Praxeolitic Aug 12 '14 at 14:35
  • 8
    no clue why people are frustrated over os.path.join behavior. In other languages, the equivalent path-join library/method behaves the exact same. It's safer, and makes more sense. – Don Cheadle Dec 04 '14 at 17:41
  • 1
    I think the example of os.path.sep should be added to this, or marked as the recommended option. This answer provides accurate information, but does not provide a solution. – rickfoosusa Feb 02 '15 at 21:23
  • 31
    This is frustrating because it's **implicit magic**, contrary to the [cardinal heuristic](https://www.python.org/dev/peps/pep-0020) of "Explicit is better than implicit." And it _is_. Language designers may believe they know better, but there exist obvious and demonstrably safe reasons to occasionally want to do this. Now we can't. This is why we can't have good things. – Cecil Curry Aug 18 '15 at 05:54
  • 3
    @mmcrae: in which other languages is it true? it is most definitely _not_ true of golang: https://golang.org/pkg/path/#Join Expand the example and click 'run'. I also don't understand how python's way can be considered safer... user input with leading slashes could result in referencing critical system files. If leading slashes aren't treated specially, then (at worst) the path references a deep path to a file/directory that doesn't exist. Granted, user input should be sanitized... but this function should not exacerbate the problem! – Mark Nov 21 '16 at 17:27
  • @DonCheadle which language? I just tried with nodejs and I didn't see this behavior – MuhsinFatih Oct 09 '20 at 10:52
  • 2
    What is a valid (preferably common) scenario where one would want a later argument in `os.path.join` to override a former path part (since a leading slash effectively removes all prior arguments). – chris Dec 23 '20 at 16:19
  • 1
    Not allowing / after the initial absolute paths (and requiring only relative paths afterwards) is not bad, but Python should throw an exception - rather than silently changing the path. (The use case of "I might want to override the initial path I provided with a following absolute path argument ia a bad choice for a method supposed to join paths). – Hejazzman Feb 05 '21 at 16:55
  • 2
    @chris you've got your `pwd`, somebody (user) gives you the path of a file, that may well be a relative path to the pwd, or an absolute path. What you want is for `os.path.join( pwd, path)` to ignore the pwd if path is a relative path, and to append it otherwise. This is probably the scenario the developers had in mind – tbrugere May 17 '21 at 08:08
170

The idea of os.path.join() is to make your program cross-platform (linux/windows/etc).

Even one slash ruins it.

So it only makes sense when being used with some kind of a reference point like os.environ['HOME'] or os.path.dirname(__file__).

Antony Hatchkins
  • 31,947
  • 10
  • 111
  • 111
93

os.path.join() can be used in conjunction with os.path.sep to create an absolute rather than relative path.

os.path.join(os.path.sep, 'home','build','test','sandboxes',todaystr,'new_sandbox')
wim
  • 338,267
  • 99
  • 616
  • 750
ghammond
  • 1,052
  • 7
  • 4
  • 11
    The use of `os.path.sep` as a first element to build an absolute path is better than any other answer here! The whole point of using `os.path` rather than basic str methods is to avoid writing `/`. Putting every subdirectory as a new argument and removing all slashes is also great. It would probably be a good idea to make sure with a check that `todaystr` does not start with a slash! ;) – snooze92 Jan 23 '14 at 08:59
  • 5
    This works on windows as well (python 2.7.6). It did not intefere with 'C:\' and joined the subdirectories. – rickfoosusa Feb 02 '15 at 21:22
  • 2
    @snooze92 The only problem is that it's more verbose and much less readable. – yugr Jan 14 '22 at 03:00
  • 1
    Indeed. Trade-offs I guess ;) – snooze92 Jan 14 '22 at 14:26
34

Do not use forward slashes at the beginning of path components, except when refering to the root directory:

os.path.join('/home/build/test/sandboxes', todaystr, 'new_sandbox')

see also: http://docs.python.org/library/os.path.html#os.path.join

miku
  • 181,842
  • 47
  • 306
  • 310
32

To help understand why this surprising behavior isn't entirely terrible, consider an application which accepts a config file name as an argument:

config_root = "/etc/myapp.conf/"
file_name = os.path.join(config_root, sys.argv[1])

If the application is executed with:

$ myapp foo.conf

The config file /etc/myapp.conf/foo.conf will be used.

But consider what happens if the application is called with:

$ myapp /some/path/bar.conf

Then myapp should use the config file at /some/path/bar.conf (and not /etc/myapp.conf/some/path/bar.conf or similar).

It may not be great, but I believe this is the motivation for the absolute path behaviour.

David Wolever
  • 148,955
  • 89
  • 346
  • 502
  • 2
    Thanks! I had always hated this behavior until reading your answer! It's documented in https://docs.python.org/3.5/library/os.path.html#os.path.join, but not the motivation for it. – Eli_B Sep 07 '17 at 11:59
  • 2
    This moment when you need exactly the solution many people deem as terrible. – ashrasmun Sep 04 '19 at 20:33
  • 6
    gotta disagree, it is entirely terrible. In this case, you should not be using a naive `sys.argv` input to determine whether to prepend the `config_root` or not. All `os.path.join` should be concerned with is joining file path elements. – user5359531 Sep 01 '20 at 20:53
  • 2
    the problem is that the name is badly chosen. what `os.path.join(p1, p2)` does is not really joining `p1` and `p2`, it's taking `p2` relatively to `p1` (or absolutely if `p2` is an absolute path) – tbrugere May 17 '21 at 08:12
  • So a specific use case should determine how a function behaves for all other use cases? Makes no sense. – Gulzar Nov 08 '21 at 08:39
  • It does make sense, and the name is fine as long as you're not assuming it will behave like `str.join`. That's how you join an absolute path, because that's exactly what it means for it to be an _absolute_ path. Perhaps it helps to think of it as where you'd end up after a chain of `cd` commands, not as a string joining operation. – wim Jun 07 '23 at 21:17
  • thanks. makes sense. but it was unintuitive as a beginner. – Sujay Phadke Jul 18 '23 at 07:54
16

It's because your '/new_sandbox/' begins with a / and thus is assumed to be relative to the root directory. Remove the leading /.

Amber
  • 507,862
  • 82
  • 626
  • 550
16

Try combo of split("/") and * for strings with existing joins.

import os

home = '/home/build/test/sandboxes/'
todaystr = '042118'
new = '/new_sandbox/'

os.path.join(*home.split("/"), todaystr, *new.split("/"))


How it works...

split("/") turns existing path into list: ['', 'home', 'build', 'test', 'sandboxes', '']

* in front of the list breaks out each item of list its own parameter

openwonk
  • 14,023
  • 7
  • 43
  • 39
10

To make your function more portable, use it as such:

os.path.join(os.sep, 'home', 'build', 'test', 'sandboxes', todaystr, 'new_sandbox')

or

os.path.join(os.environ.get("HOME"), 'test', 'sandboxes', todaystr, 'new_sandbox')
NuclearPeon
  • 5,743
  • 4
  • 44
  • 52
4

Try with new_sandbox only

os.path.join('/home/build/test/sandboxes/', todaystr, 'new_sandbox')
Remi Guan
  • 21,506
  • 17
  • 64
  • 87
YOU
  • 120,166
  • 34
  • 186
  • 219
4

do it like this, without too the extra slashes

root="/home"
os.path.join(root,"build","test","sandboxes",todaystr,"new_sandbox")
ghostdog74
  • 327,991
  • 56
  • 259
  • 343
1
os.path.join("a", *"/b".split(os.sep))
'a/b'

a fuller version:

import os

def join (p, f, sep = os.sep):
    f = os.path.normpath(f)
    if p == "":
        return (f);
    else:
        p = os.path.normpath(p)
        return (os.path.join(p, *f.split(os.sep)))

def test (p, f, sep = os.sep):
    print("os.path.join({}, {}) => {}".format(p, f, os.path.join(p, f)))
    print("        join({}, {}) => {}".format(p, f, join(p, f, sep)))

if __name__ == "__main__":
    # /a/b/c for all
    test("\\a\\b", "\\c", "\\") # optionally pass in the sep you are using locally
    test("/a/b", "/c", "/")
    test("/a/b", "c")
    test("/a/b/", "c")
    test("", "/c")
    test("", "c")
Goblinhack
  • 2,859
  • 1
  • 26
  • 26
  • What if os.sep is actually `"\"`? Then your first example becomes `os.path.join("a", *"/b".split("\\"))`, which yields `"/b"`... I doubt that's the intended result. – NichtJens May 16 '20 at 22:13
  • 1
    Updated - I suppose you have to give a hint as the path sep you are using locally independent of that of the os you are running on – Goblinhack May 17 '20 at 01:54
  • 1
    Yes. Alternatively one could split on both commonly used options ... but then some other OS could come up with a third. – NichtJens May 17 '20 at 09:30
0

Note that a similar issue can bite you if you use os.path.join() to include an extension that already includes a dot, which is what happens automatically when you use os.path.splitext(). In this example:

components = os.path.splitext(filename)
prefix = components[0]
extension = components[1]
return os.path.join("avatars", instance.username, prefix, extension)

Even though extension might be .jpg you end up with a folder named "foobar" rather than a file called "foobar.jpg". To prevent this you need to append the extension separately:

return os.path.join("avatars", instance.username, prefix) + extension
shacker
  • 14,712
  • 8
  • 89
  • 89
0

you can strip the '/':

>>> os.path.join('/home/build/test/sandboxes/', todaystr, '/new_sandbox/'.strip('/'))
'/home/build/test/sandboxes/04122019/new_sandbox'
suhailvs
  • 20,182
  • 14
  • 100
  • 98
0

The problem is your laptop maybe running Window. And Window annoyingly use back lash instead of forward slash'/'.
To make your program cross-platform (linux/windows/etc). You shouldn't provide any slashes (forward or backward) in your path if you want os.path.join to handle them properly. you should using:

os.path.join(os.environ.get("HOME"), 'test', 'sandboxes', todaystr, 'new_sandbox')

Or throw some Path(__file__).resolve().parent (path to parent of current file) or anything so that you don't use any slash inside os.path.join

Trọng Nhân
  • 39
  • 8
0

Please refer following code snippet for understanding os.path.join(a, b)

a = '/home/user.name/foo/'
b = '/bar/file_name.extension'

print(os.path.join(a, b))
>>> /bar/file_name.extension

OR

a = '/home/user.name/foo'
b = '/bar/file_name.extension'
print(os.path.join(a, b))
>>> /bar/file_name.extension

But, when

a = '/home/user.name/foo/'
b = 'bar/file_name.extension'

print(os.path.join(a, b))
>>> /bar/file_name.extension

OR

a = '/home/user.name/foo'
b = 'bar/file_name.extension'
print(os.path.join(a, b))
>>> /home/user.name/foo/bar/file_name.extension
afghani
  • 467
  • 5
  • 7
-1

I'd recommend to strip from the second and the following strings the string os.path.sep, preventing them to be interpreted as absolute paths:

first_path_str = '/home/build/test/sandboxes/'
original_other_path_to_append_ls = [todaystr, '/new_sandbox/']
other_path_to_append_ls = [
    i_path.strip(os.path.sep) for i_path in original_other_path_to_append_ls
]
output_path = os.path.join(first_path_str, *other_path_to_append_ls)
Dr Fabio Gori
  • 1,105
  • 16
  • 21