How to extract domain name from url?

Question

How do I extract the domain name from a url using bash? like: http://example.com/ to example.com must work for any tld, not just .com

Answer 1

You can use simple AWK way to extract the domain name as follows:

echo http://example.com/index.php | awk -F[/:] '{print $4}'

OUTPUT: example.com

:-)

Answer 2

$ URI="http://user:pw@example.com:80/"
$ echo $URI | sed -e 's/[^/]*\/\/\([^@]*@\)\?\([^:/]*\).*/\2/'
example.com

see http://en.wikipedia.org/wiki/URI_scheme

Answer 3

basename "http://example.com"

Now of course, this won't work with a URI like this: http://www.example.com/index.html but you could do the following:

basename $(dirname "http://www.example.com/index.html")

Or for more complex URIs:

echo "http://www.example.com/somedir/someotherdir/index.html" | cut -d'/' -f3

-d means "delimiter" and -f means "field"; in the above example, the third field delimited by the forward slash '/' is www.example.com.

Answer 4

echo $URL | cut -d'/' -f3 | cut -d':' -f1

Works for URLs:

http://host.example.com
http://host.example.com/hi/there
http://host.example.com:2345/hi/there
http://host.example.com:2345

Answer 5

sed -E -e 's_.*://([^/@]*@)?([^/:]+).*_\2_'

e.g.

$ sed -E -e 's_.*://([^/@]*@)?([^/:]+).*_\2_' <<< 'http://example.com'
example.com

$ sed -E -e 's_.*://([^/@]*@)?([^/:]+).*_\2_' <<< 'https://example.com'
example.com

$ sed -E -e 's_.*://([^/@]*@)?([^/:]+).*_\2_' <<< 'http://example.com:1234/some/path'
example.com

$ sed -E -e 's_.*://([^/@]*@)?([^/:]+).*_\2_' <<< 'http://user:pass@example.com:1234/some/path'
example.com

$ sed -E -e 's_.*://([^/@]*@)?([^/:]+).*_\2_' <<< 'http://user:pass@example.com:1234/some/path#fragment'
example.com

$ sed -E -e 's_.*://([^/@]*@)?([^/:]+).*_\2_' <<< 'http://user:pass@example.com:1234/some/path#fragment?params=true'
example.com

Answer 6

#!/usr/bin/perl -w
use strict;

my $url = $ARGV[0];

if($url =~ /([^:]*:\/\/)?([^\/]+\.[^\/]+)/g) {
  print $2;
}

Usage:

./test.pl 'https://example.com'
example.com

./test.pl 'https://www.example.com/'
www.example.com

./test.pl 'example.org/'
example.org

 ./test.pl 'example.org'
example.org

./test.pl 'example'  -> no output

And if you just want the domain and not the full host + domain use this instead:

#!/usr/bin/perl -w
use strict;

my $url = $ARGV[0];
if($url =~ /([^:]*:\/\/)?([^\/]*\.)*([^\/\.]+\.[^\/]+)/g) {
  print $3;
}

Answer 7

Instead of using regex to do this you can use python's urlparse:

 URL=http://www.example.com

 python -c "from urlparse import urlparse
 url = urlparse('$URL')
 print url.netloc"

You could either use it like this or put it in a small script. However this still expects a valid scheme identifier, looking at your comment your input doesn't necessarily provide one. You can specify a default scheme, but urlparse expects the netloc to start with '//' :

url = urlparse('//www.example.com/index.html','http')

So you will have to prepend those manually, i.e:

 python -c "from urlparse import urlparse
 if '$URL'.find('://') == -1 then:
   url = urlparse('//$URL','http')
 else:
   url = urlparse('$URL')
 print url.netloc"

Answer 8

3 answers: short URL parsing (shell+bash) and full TLD extractor

Remark about question:

Question stand for regex, but the goal there is to split string on / character!! XY problem, using regex for this kind of job is overkill!

Posix shell first

Instead of using forks to another binaries, like awk, perl, cut or else, we could use parameter expansions which is quicker:

URL="http://example.com/some/path/to/page.html"
prot="${URL%%:*}"
link="${URL#$prot://}"
domain="${link%%/*}"
link="${link#$domain}"
printf '%-8s: %s\n' Protocol "${prot%:}" Domain "$domain" Link "$link"

Protocol: http
Domain  : example.com
Link    : /some/path/to/page.html

Note: This work even with file URL:

URL=file:///tmp/so/test.xml 
prot="${URL%%:*}"
link="${URL#$prot://}"
domain="${link%%/*}"
link="${link#$domain}"
printf '%-8s: %s\n' Protocol "${prot%:}" Domain "$domain" Link "$link"

Protocol: file
Domain  : 
Link    : /tmp/so/test.xml

read url parts using bash

As this question is tagged bash and no answer address read short, quick and reliable solution:

URL="http://example.com/some/path/to/page.html"

IFS=/ read -r prot _ domain link <<<"$URL"

That's all. As read is a builtin, this is the quickest way!! (** See comment)

From there you could

printf '%-8s: %s\n' Protocol "${prot%:}" Domain "$domain" Link "/$link"

Protocol: http
Domain  : example.com
Link    : /some/path/to/page.html

You could even check for port:

URL="http://example.com:8000/some/path/to/page.html"
IFS=/ read -r prot _ domain link <<<"$URL"
IFS=: read -r domain port <<<"$domain"

printf '%-8s: %s\n' Protocol "${prot%:}" Domain "$domain" Port "$port" Link "/$link"

Protocol: http
Domain  : example.com
Port    : 8000
Link    : /some/path/to/page.html

Full parsing with default ports:

URL="https://stackoverflow.com/questions/2497215/how-to-extract-domain-name-from-url"
declare -A DEFPORTS='([http]=80 [https]=443 [ipp]=631 [ftp]=21)'
IFS=/ read -r prot _ domain link <<<"$URL"
IFS=: read -r domain port <<<"$domain"

printf '%-8s: %s\n' Protocol "${prot%:}" Domain "$domain" \
    Port  "${port:-${DEFPORTS[${prot%:}]}}" Link "/$link"

Protocol: https
Domain  : stackoverflow.com
Port    : 443
Link    : /questions/2497215/how-to-extract-domain-name-from-url

Full Top Level Domain extractor (in pure bash):

Regarding public suffix and @tripleee' comment

There is one fork to wget which is done only once, at function initialization:

declare -A TLD='()'
initTld () { 
    local tld
    while read -r tld; do
        [[ -n ${tld//*[ \/;*]*} ]] && TLD["${tld#\!}"]=''
    done < <(
      wget -qO - https://publicsuffix.org/list/public_suffix_list.dat
    )
}
tldExtract () { 
    if [[ $1 == -v ]] ;then local _tld_out_var=$2;shift 2;fi
    local dom tld=$1 _tld_out_var
    while [[ ! -v TLD[${tld}] ]] && [[ -n $tld ]]; do
        IFS=. read -r dom tld <<< "$tld"
    done
    if [[ -v _tld_out_var ]] ;then
        printf -v $_tld_out_var '%s %s' "$dom" "$tld"
    else
        echo "$dom $tld"
    fi
}
initTld ; unset -f initTld

Then

tldExtract www.stackoverflow.com
stackoverflow com

tldExtract sub.www.test.co.uk
test co.uk

tldExtract -v myVar sub.www.test.co.uk
echo ${myVar% *}
test
echo ${myVar#* }
co.uk

tldExtract -v myVar www2.sub.city.nagoya.jp
echo $myVar 
sub city.nagoya.jp

Answer 9

there is so little info on how you get those urls...please show more info next time. are there parameters in the url etc etc... Meanwhile, just simple string manipulation for your sample url

eg

$ s="http://example.com/index.php"
$ echo ${s/%/*}  #get rid of last "/" onwards
http://example.com
$ s=${s/%\//}  
$ echo ${s/#http:\/\//} # get rid of http://
example.com

other ways, using sed(GNU)

$ echo $s | sed 's/http:\/\///;s|\/.*||'
example.com

use awk

$ echo $s| awk '{gsub("http://|/.*","")}1'
example.com

Answer 10

The following will output "example.com":

URI="http://user@example.com/foo/bar/baz/?lala=foo" 
ruby -ruri -e "p URI.parse('$URI').host"

For more info on what you can do with Ruby's URI class you'd have to consult the docs.

Answer 11

One solution that would cover for more cases would be based on sed regexps:

echo http://example.com/index.php | sed -e 's#^https://\|^http://##' -e 's#:.*##' -e 's#/.*##'

That would work for URLs like: http://example.com/index.php, http://example.com:4040/index.php, https://example.com/index.php

Answer 12

Please note that extracting domain-name only from a URL is a bit tricky because domain name place in the hostname depends on the country (or more generally on the TLD) being used.

eg. for Argentina: www.personal.com.ar Domain name is personal.com.ar, not com.ar because this TLD uses subzones to specify type of organization.

The tool that I've found to manage well these cases is tldextract

So based on the FQDN (host part of the URL), you would get the domain reliably this way:

tldextract personal.com.ar | cut -d " " -f 2,3 | sed 's/ /./'

(the other answers to get the FQDN out of the URL are good and should be used)

hope this helps :) and thanks to tripleee !

Answer 13

With Ruby you can use the Domainatrix library / gem

http://www.pauldix.net/2009/12/parse-domains-from-urls-easily-with-domainatrix.html

require 'rubygems'
require 'domainatrix'
s = 'http://www.champa.kku.ac.th/dir1/dir2/file?option1&option2'
url = Domainatrix.parse(s)
url.domain
=> "kku"

great tool! :-)

Answer 14

Here's the node.js way, it works with or without ports and deep paths:

//get-hostname.js
'use strict';

const url = require('url');
const parts = url.parse(process.argv[2]);

console.log(parts.hostname);

Can be called like:

node get-hostname.js http://foo.example.com:8080/test/1/2/3.html
//foo.example.com

Docs: https://nodejs.org/api/url.html

Answer 15

Pure Bash implementation without any sub-shell or sub-process:

# Extract host from an URL
#   $1: URL
function extractHost {
    local s="$1"
    s="${s/#*:\/\/}" # Parameter Expansion & Pattern Matching
    echo -n "${s/%+(:*|\/*)}"
}

E.g. extractHost "docker://1.2.3.4:1234/a/v/c" will output 1.2.3.4

Answer 16

Using bash built-in regex (no external utilities needed):

#!/usr/bin/env bash

url=https://stackoverflow.com/questions/2497215/how-to-extract-domain-name-from-url

if [[ $url =~ ^(https?://[^/]+) ]]; then
  host="${BASH_REMATCH[1]}"
  echo "HOST: $host"
else
  echo "Invalid URL $url"
  exit 1
fi

# OUTPUT
# HOST: https://stackoverflow.com

See also:

• https://www.gnu.org/software/bash/manual/html_node/Bash-Variables.html
• http://molk.ch/tips/gnu/bash/rematch.html