How can I split a URL string up into separate parts in Python?

Question

I decided that I'll learn Python tonight :) I know C pretty well (wrote an OS in it), so I'm not a noob in programming, so everything in Python seems pretty easy, but I don't know how to solve this problem: let's say I have this address:

http://example.com/random/folder/path.html

Now how can I create two strings from this, one containing the "base" name of the server, so in this example it would be

http://example.com/

and another containing the thing without the last filename, so in this example it would be

http://example.com/random/folder/

Also I of course know the possibility to just find the third and last slash respectively, but is there a better way?

Also it would be cool to have the trailing slash in both cases, but I don't care since it can be added easily. So is there a good, fast, effective solution for this? Or is there only "my" solution, finding the slashes?

Answer 1

The urlparse module in Python 2.x (or urllib.parse in Python 3.x) would be the way to do it.

>>> from urllib.parse import urlparse
>>> url = 'http://example.com/random/folder/path.html'
>>> parse_object = urlparse(url)
>>> parse_object.netloc
'example.com'
>>> parse_object.path
'/random/folder/path.html'
>>> parse_object.scheme
'http'
>>>

If you wanted to do more work on the path of the file under the URL, you can use the posixpath module:

>>> from posixpath import basename, dirname
>>> basename(parse_object.path)
'path.html'
>>> dirname(parse_object.path)
'/random/folder'

After that, you can use posixpath.join to glue the parts together.

Note: Windows users will choke on the path separator in os.path. The posixpath module documentation has a special reference to URL manipulation, so all's good.

Answer 2

If this is the extent of your URL parsing, Python's inbuilt rpartition will do the job:

>>> URL = "http://example.com/random/folder/path.html"
>>> Segments = URL.rpartition('/')
>>> Segments[0]
'http://example.com/random/folder'
>>> Segments[2]
'path.html'

From Pydoc, str.rpartition:

Splits the string at the last occurrence of sep, and returns a 3-tuple containing the part before the separator, the separator itself, and the part after the separator. If the separator is not found, return a 3-tuple containing two empty strings, followed by the string itself

What this means is that rpartition does the searching for you, and splits the string at the last (right most) occurrence of the character you specify (in this case / ). It returns a tuple containing:

(everything to the left of char , the character itself , everything to the right of char)

Answer 3

I have no experience with Python, but I found the urlparse module, which should do the job.

Answer 4

In Python, a lot of operations are done using lists. The urlparse module mentioned by Sebasian Dietz may well solve your specific problem, but if you're generally interested in Pythonic ways to find slashes in strings, for example, try something like this:

url = 'http://example.com/random/folder/path.html'

# Create a list of each bit between slashes
slashparts = url.split('/')

# Now join back the first three sections 'http:', '' and 'example.com'
basename = '/'.join(slashparts[:3]) + '/'

# All except the last one
dirname = '/'.join(slashparts[:-1]) + '/'

print 'slashparts = %s' % slashparts
print 'basename = %s' % basename
print 'dirname = %s' % dirname

The output of this program is this:

slashparts = ['http:', '', 'example.com', 'random', 'folder', 'path.html']
basename = http://example.com/
dirname = http://example.com/random/folder/

The interesting bits are split, join, the slice notation array[A:B] (including negatives for offsets-from-the-end) and, as a bonus, the % operator on strings to give printf-style formatting.

Answer 5

It seems like the posixpath module mentioned in sykora's answer is not available in my Python setup (Python 2.7.3).

As per this article, it seems that the "proper" way to do this would be using...

• urlparse.urlparse and urlparse.urlunparse can be used to detach and reattach the base of the URL
• The functions of os.path can be used to manipulate the path
• urllib.url2pathname and urllib.pathname2url (to make path name manipulation portable, so it can work on Windows and the like)

So for example (not including reattaching the base URL)...

>>> import urlparse, urllib, os.path
>>> os.path.dirname(urllib.url2pathname(urlparse.urlparse("http://example.com/random/folder/path.html").path))
'/random/folder'

Answer 6

You can use Python's library furl:

f = furl.furl("http://example.com/random/folder/path.html")
print(str(f.path))  # '/random/folder/path.html'
print(str(f.path).split("/")) # ['', 'random', 'folder', 'path.html']

To access word after first "/", use:

str(f.path).split("/") # 'random'