Get protocol + host name from URL

Question

In my Django app, I need to get the host name from the referrer in request.META.get('HTTP_REFERER') along with its protocol so that from URLs like:

https://docs.google.com/spreadsheet/ccc?key=blah-blah-blah-blah#gid=1
https://stackoverflow.com/questions/1234567/blah-blah-blah-blah
http://www.example.com
https://www.other-domain.example/whatever/blah/blah/?v1=0&v2=blah+blah

I should get:

https://docs.google.com/
https://stackoverflow.com/
http://www.example.com
https://www.other-domain.example/

I looked over other related questions and found about urlparse, but that didn't do the trick since

>>> urlparse(request.META.get('HTTP_REFERER')).hostname
'docs.google.com'

score 361 · Accepted Answer · edited Jul 22 '18 at 15:35

361

You should be able to do it with urlparse (docs: python2, python3):

from urllib.parse import urlparse
# from urlparse import urlparse  # Python 2
parsed_uri = urlparse('http://stackoverflow.com/questions/1234567/blah-blah-blah-blah' )
result = '{uri.scheme}://{uri.netloc}/'.format(uri=parsed_uri)
print(result)

# gives
'http://stackoverflow.com/'

edited Jul 22 '18 at 15:35

wim

338,267
99
616
750

answered Mar 08 '12 at 23:17

kgr

9,750
2
38
43

this answer *adds* a `/` to the third example `http://www.domain.com`, but I think this might be a shortcoming of the question, not of the answer. – SingleNegationElimination Mar 09 '12 at 01:36
@TokenMacGuy: ya, my bad... didn't notice the missing **/** – Gerard Mar 09 '12 at 03:16
15

I don't think this is a good solution, as `netloc` is not domain: try `urlparse.urlparse('http://user:pass@example.com:8080')` and find it gives parts like `'user:pass@'` and `':8080'` – starrify Oct 21 '14 at 08:02
27

The urlparse module is renamed to urllib.parse in Python 3. So, `from urllib.parse import urlparse` – SparkAndShine Jul 21 '15 at 21:37
This answers what the author meant to ask, but not what was actually stated. For those looking for domain name and not hostname (as this solution provides) I suggest looking at dm03514's answer that is currently below. Python's urlparse cannot give you domain names. Something that seems an oversight. – Scone Jul 18 '18 at 18:27
String operations should be avoided at allcosts. use built ins from urllib.parse: https://stackoverflow.com/a/32214840/8478 – Marc Apr 17 '23 at 19:32

dm03514 · Answer 2 · 2015-06-03T12:57:26.280

https://github.com/john-kurkowski/tldextract

This is a more verbose version of urlparse. It detects domains and subdomains for you.

From their documentation:

>>> import tldextract
>>> tldextract.extract('http://forums.news.cnn.com/')
ExtractResult(subdomain='forums.news', domain='cnn', suffix='com')
>>> tldextract.extract('http://forums.bbc.co.uk/') # United Kingdom
ExtractResult(subdomain='forums', domain='bbc', suffix='co.uk')
>>> tldextract.extract('http://www.worldbank.org.kg/') # Kyrgyzstan
ExtractResult(subdomain='www', domain='worldbank', suffix='org.kg')

ExtractResult is a namedtuple, so it's simple to access the parts you want.

>>> ext = tldextract.extract('http://forums.bbc.co.uk')
>>> ext.domain
'bbc'
>>> '.'.join(ext[:2]) # rejoin subdomain and domain
'forums.bbc'

This is the correct answer for the question as written, how to get the DOMAIN name. The chosen solution provides the HOSTNAME, which I believe is what the author wanted in the first place. — Scone, Jul 18 '18 at 18:29

score 52 · Answer 3 · answered Dec 22 '13 at 11:20

52

Python3 using urlsplit:

from urllib.parse import urlsplit
url = "http://stackoverflow.com/questions/9626535/get-domain-name-from-url"
base_url = "{0.scheme}://{0.netloc}/".format(urlsplit(url))
print(base_url)
# http://stackoverflow.com/

answered Dec 22 '13 at 11:20

Marc SJ

711
8
12

score 34 · Answer 4 · answered Aug 25 '15 at 22:02

34

>>> import urlparse
>>> url = 'http://stackoverflow.com/questions/1234567/blah-blah-blah-blah'
>>> urlparse.urljoin(url, '/')
'http://stackoverflow.com/'

answered Aug 25 '15 at 22:02

png

5,990
2
25
16

4

For Python 3 the import is `from urllib.parse import urlparse`. – Jeff Bowen Nov 06 '18 at 22:22
2

The argument doesn't seem intuitive, but it works great as a very simple native solution – Sunny Patel Aug 26 '22 at 19:25

SebMa · Answer 5 · 2018-02-05T16:21:40.463

25

Pure string operations :):

>>> url = "http://stackoverflow.com/questions/9626535/get-domain-name-from-url"
>>> url.split("//")[-1].split("/")[0].split('?')[0]
'stackoverflow.com'
>>> url = "stackoverflow.com/questions/9626535/get-domain-name-from-url"
>>> url.split("//")[-1].split("/")[0].split('?')[0]
'stackoverflow.com'
>>> url = "http://foo.bar?haha/whatever"
>>> url.split("//")[-1].split("/")[0].split('?')[0]
'foo.bar'

That's all, folks.

edited Feb 05 '18 at 16:21

answered Apr 13 '16 at 21:26

SebMa

4,037
29
39

2

Good and simple option, but fails in some cases, e.g. https://foo.bar?haha – Simon Steinberger Dec 13 '17 at 22:02
1

@SimonSteinberger :-) How'bout this : `url.split("//")[-1].split("/")[0].split('?')[0]` :-)) – SebMa Feb 05 '18 at 16:16

score 15 · Answer 6 · answered Mar 03 '20 at 20:38

The standard library function urllib.parse.urlsplit() is all you need. Here is an example for Python3:

>>> import urllib.parse
>>> o = urllib.parse.urlsplit('https://user:pass@www.example.com:8080/dir/page.html?q1=test&q2=a2#anchor1')
>>> o.scheme
'https'
>>> o.netloc
'user:pass@www.example.com:8080'
>>> o.hostname
'www.example.com'
>>> o.port
8080
>>> o.path
'/dir/page.html'
>>> o.query
'q1=test&q2=a2'
>>> o.fragment
'anchor1'
>>> o.username
'user'
>>> o.password
'pass'

score 9 · Answer 7 · edited Apr 13 '19 at 07:32

9

if you think your url is valid then this will work all the time

domain = "http://google.com".split("://")[1].split("/")[0]

edited Apr 13 '19 at 07:32

Pedro Lobito

94,083
31
258
268

answered May 07 '18 at 12:57

Jeeva

3,975
3
23
47

The last `split` is wrong, there are no more forward slashes to split. – Pedro Lobito Apr 13 '19 at 05:23
2

it's won't be a problem, if there are no more slashes then, the list will return with one element. so it will work whether there is a slash or not – Jeeva Apr 13 '19 at 07:09
1

I edited your answer the be able to remove the down-vote. Nice explanation. Tks. – Pedro Lobito Apr 13 '19 at 07:32

Faiz · Answer 8 · 2017-11-23T17:03:02.153

6

Here is a slightly improved version:

urls = [
    "http://stackoverflow.com:8080/some/folder?test=/questions/9626535/get-domain-name-from-url",
    "Stackoverflow.com:8080/some/folder?test=/questions/9626535/get-domain-name-from-url",
    "http://stackoverflow.com/some/folder?test=/questions/9626535/get-domain-name-from-url",
    "https://StackOverflow.com:8080?test=/questions/9626535/get-domain-name-from-url",
    "stackoverflow.com?test=questions&v=get-domain-name-from-url"]
for url in urls:
    spltAr = url.split("://");
    i = (0,1)[len(spltAr)>1];
    dm = spltAr[i].split("?")[0].split('/')[0].split(':')[0].lower();
    print dm

Output

stackoverflow.com
stackoverflow.com
stackoverflow.com
stackoverflow.com
stackoverflow.com

Fiddle: https://pyfiddle.io/fiddle/23e4976e-88d2-4757-993e-532aa41b7bf0/?i=true

edited Nov 23 '17 at 17:03

answered Nov 23 '17 at 11:02

Faiz

5,331
10
45
57

IMHO the best solution, because simple and it considers all sorts of rare cases. Thanks! – Simon Steinberger Dec 13 '17 at 22:03
2

neither simple nor improved – Corey Goldberg Aug 12 '18 at 22:54
This is not a solution for the question because you do not provide protocol (https:// or http://) – Nairum Oct 15 '19 at 14:31

score 5 · Answer 9 · answered Jun 07 '13 at 22:06

Is there anything wrong with pure string operations:

url = 'http://stackoverflow.com/questions/9626535/get-domain-name-from-url'
parts = url.split('//', 1)
print parts[0]+'//'+parts[1].split('/', 1)[0]
>>> http://stackoverflow.com

If you prefer having a trailing slash appended, extend this script a bit like so:

parts = url.split('//', 1)
base = parts[0]+'//'+parts[1].split('/', 1)[0]
print base + (len(url) > len(base) and url[len(base)]=='/'and'/' or '')

That can probably be optimized a bit ...

it's not wrong but we got a tool that already does the work, let's not reinvent the wheel ;) — Gerard, Jun 12 '13 at 02:33

Orix Au Yeung · Answer 10 · 2018-11-02T03:51:41.677

3

I know it's an old question, but I too encountered it today. Solved this with an one-liner:

import re
result = re.sub(r'(.*://)?([^/?]+).*', '\g<1>\g<2>', url)

edited Nov 02 '18 at 03:51

answered Nov 02 '18 at 03:32

Orix Au Yeung

111
6

score 3 · Answer 11 · answered Nov 05 '19 at 21:49

3

It could be solved by re.search()

import re
url = 'https://docs.google.com/spreadsheet/ccc?key=blah-blah-blah-blah#gid=1'
result = re.search(r'^http[s]*:\/\/[\w\.]*', url).group()
print(result)

#result
'https://docs.google.com'

answered Nov 05 '19 at 21:49

kiwi

31
1

This answer was helpful for my case. Thanks. – aysebilgegunduz Apr 08 '22 at 21:34
Does not include port – biscuit314 Jun 30 '23 at 16:55

Mirko · Answer 12 · 2020-09-25T01:52:22.600

2

You can simply use urljoin with relative root '/' as second argument:

import urllib.parse


url = 'https://stackoverflow.com/questions/9626535/get-protocol-host-name-from-url'
root_url = urllib.parse.urljoin(url, '/')
print(root_url)

edited Sep 25 '20 at 01:52

answered Jul 15 '20 at 18:47

Mirko

2,231
2
21
17

score 2 · Answer 13 · answered Dec 21 '21 at 17:42

2

This is the simple way to get the root URL of any domain.

from urllib.parse import urlparse

url = urlparse('https://stackoverflow.com/questions/9626535/')
root_url = url.scheme + '://' + url.hostname
print(root_url) # https://stackoverflow.com

answered Dec 21 '21 at 17:42

Praveen Kumar

849
8
8

score 2 · Answer 14 · answered Mar 09 '12 at 01:43

This is a bit obtuse, but uses urlparse in both directions:

import urlparse
def uri2schemehostname(uri):
    urlparse.urlunparse(urlparse.urlparse(uri)[:2] + ("",) * 4)

that odd ("",) * 4 bit is because urlparse expects a sequence of exactly len(urlparse.ParseResult._fields) = 6

score 0 · Answer 15 · answered Feb 25 '19 at 18:34

to get domain/hostname and Origin*

url = 'https://stackoverflow.com/questions/9626535/get-protocol-host-name-from-url'
hostname = url.split('/')[2] # stackoverflow.com
origin = '/'.join(url.split('/')[:3]) # https://stackoverflow.com

*Origin is used in XMLHttpRequest headers

Juraj · Answer 16 · 2018-06-22T03:21:14.430

-1

If it contains less than 3 slashes thus you've it got and if not then we can find the occurrence between it:

import re

link = http://forum.unisoftdev.com/something

slash_count = len(re.findall("/", link))
print slash_count # output: 3

if slash_count > 2:
   regex = r'\:\/\/(.*?)\/'
   pattern  = re.compile(regex)
   path = re.findall(pattern, url)

   print path

edited Jun 22 '18 at 03:21

answered Jun 21 '18 at 20:55

Juraj

62
3

Get protocol + host name from URL

16 Answers16

Python3 using urlsplit:

Linked

Related