Regex to pull last 2 segments from FQDN

Question

Working on trying to figure out some regex to pull out the last 2 segments of an FQDN.

^.*\shostname=[\w-]+\.(?P<myfield>[^\t]+)

This RegEx works and takes out the first segment of an FQDN.

www.aaa.bbb.someurl.net --> aaa.bbb.someurl.net

But… I only want to keep the last 2 segments of any FQDN.

I need it to be --> someurl.net

Other restrictions:
The hostname field will always be at least 3 segments - don't know the max.

This is for Splunk so I can't use a script. I need it to be PCRE compatible regex.

Here is an example of data:

2021-07-20 18:19:14 reason=Not allowed to browse this category event_id=12345 protocol=HTTP action=Blocked transactionsize=16051 responsesize=789 requestsize=456 urlcategory=Blocked serverip=1.2.4.5 clienttranstime=0 requestmethod=GET refererURL=None useragent=Microsoft-Delivery location=Internal ClientIP=5.6.7.8 status=403 user=John url=dl.delivery.mp.microsoft.com/filestreamingservice/files/abcd-efgh-ijkl/pieceshash vendor=Zscaler hostname=dl.delivery.mp.microsoft.com

From this I data I need the field “myfield” to be: microsoft.com.

What if it's www.amazon.co.uk? – Wai Ha Lee Jul 20 '21 at 20:48 — Wai Ha Lee, Jul 20 '21 at 20:48

Wiktor Stribiżew · Accepted Answer · 2021-07-21T14:18:14.480

2

The original answer with a much simpler regex ((?:\s|^)hostname=(?:[^\s.]+\.)*(?P<myfield>[^\s.]+\.[^\s.]+)) that worked for OP is in the question history.

You can use

(?:\s|^)hostname=(?:[^\s.]+\.)*?(?P<myfield>[^\s.]+\.(?:(?:ac|co)\.uk|govt?\.uk|judiciary\.uk|l(?:ea|td)\.uk|m(?:e|il|od)\.uk|n(?:et|hs|ic)\.uk|orgn?\.uk|p(?:arliament|lc|olice)\.uk|(?:royal|sch)\.uk|[^\s.]+)(?!\S))

Or, to match the last hostname=... value on a line:

^.*\shostname=(?:[^\s.]+\.)*?(?P<myfield>[^\s.]+\.(?:(?:ac|co)\.uk|govt?\.uk|judiciary\.uk|l(?:ea|td)\.uk|m(?:e|il|od)\.uk|n(?:et|hs|ic)\.uk|orgn?\.uk|p(?:arliament|lc|olice)\.uk|(?:royal|sch)\.uk|[^\s.]+)(?!\S))

See the regex #1 demo and regex #2 demo. Details:

(?:\s|^) - either a whitespace or start of string
hostname= - a literal substring
(?:[^\s.]+\.)*? - zero or more (but as few as possible) occurrences of one or more chars other than whitespace and dot and then a dot
(?P<myfield>[^\s.]+\.(?:(?:ac|co)\.uk|govt?\.uk|judiciary\.uk|l(?:ea|td)\.uk|m(?:e|il|od)\.uk|n(?:et|hs|ic)\.uk|orgn?\.uk|p(?:arliament|lc|olice)\.uk|(?:royal|sch)\.uk|[^\s.]+)(?!\S)) - Group "myfield": one or more chars other than whitespace and dot, then a dot, then any second-level domain or any one or more chars other than whitespace and dot and then either a whitespace or end of string.

.ac.uk
.co.uk
.gov.uk
.judiciary.uk
.ltd.uk
.me.uk
.mod.uk
.net.uk
.nhs.uk
.nic.uk
.org.uk
.parliament.uk
.plc.uk
.police.uk
.royal.uk
.sch.uk
.co.uk
.ltd.uk
.me.uk
.net.uk
.nic.uk
.org.uk
.plc.uk
.sch.uk
.govt.uk
.orgn.uk
.lea.uk
.mil.uk

If you want to add more second-level domain names, add more to the list and use https://www.myregextester.com or suchlike services to built the word list regex.

edited Jul 21 '21 at 14:18

answered Jul 20 '21 at 20:42

Wiktor Stribiżew

607,720
39
448
563

this is neat and all ... but doesn't account for things like www.amazon.co.uk - all you'll get back is "co.uk" – warren Jul 21 '21 at 13:36
@warren It is not a requirement in the question. – Wiktor Stribiżew Jul 21 '21 at 13:40
@WiktorStribiżew - you should still handle it: it's a far too common use case to ignore. OP probably didn't think of it initially, but it was left as a comment on OP – warren Jul 21 '21 at 13:42
also - this regex `hostname=.+\.(?[\w_-]+\.[\w_-]+$)` is a 48-step match vs your 1396-step match :) (https://regex101.com/r/asdkQg/1 vs https://regex101.com/r/5gtAVP/1) – warren Jul 21 '21 at 13:48
@warren You are not accounting for the fact that my regex matches anywhere in the string while you assume the `hostname` is at the end of string. This is premature optimization and inappropriate comparison of different patterns. – Wiktor Stribiżew Jul 21 '21 at 13:51
@warren Updated. And please do not compare pears to oranges. – Wiktor Stribiżew Jul 21 '21 at 13:56
@WiktorStribiżew - in OP's sample data, he clearly shows `hostname=...` to be at the end. And it's not a "premature optimization" - this is Splunk Regex 101 :) – warren Jul 21 '21 at 14:01
I don't have any .co.Countrycode so I wasn't worried about that. The edited version is cool but is way too cluttered for me. The original works perfectly: (?:\s|^)hostname=(?:[^\s.]+\.)*(?P[^\s.]+\.[^\s.]+) – Ben Jul 21 '21 at 14:06
@Ben Sorry, I had to remove it due to other users' concerns. It is still available in the [question history](https://stackoverflow.com/revisions/68461091/1). – Wiktor Stribiżew Jul 21 '21 at 14:17

score 1 · Answer 2 · answered Jul 20 '21 at 20:43

You could match all following non whitspace chars after hostname= and then use a capture group to capture the last part with a single dot.

^.*\shostname=(?:\S+\.)?([^\s.]+\.[^\s.]+)

^.*\shostname=
(?:\S+\.)? Optionally match a possible dot before
( Capture group 1
- [^\s.]+\.[^\s.]+ Match 2 non dot parts with a . in between
) Close group

Regex demo

score -1 · Answer 3 · edited Jul 20 '21 at 21:46

If you would like to account for country codes, I've previously answered this at: Get Domain Extension From Hostname

The regular expression would look something like (shortened version): \w+((\.[a-z]{2,3})(\.(uk|au))?)$

The full expression with all country codes: \w+((\.[a-z]{2,3})(\.(ad|ae|af|ag|ai|al|am|ao|aq|ar|as|at|au|aw|ax|az|ba|bb|bd|be|bf|bg|bh|bi|bj|bl|bm|bn|bo|bq|br|bs|bt|bv|bw|by|bz|ca|cc|cd|cf|cg|ch|ci|ck|cl|cm|cn|co|cr|cu|cv|cw|cx|cy|cz|de|dj|dk|dm|do|dz|ec|ee|eg|er|es|et|fi|fj|fk|fm|fo|fr|ga|gb|gd|ge|gf|gg|gh|gi|gl|gm|gn|gp|gq|gr|gs|gt|gu|gw|gy|hk|hm|hn|hr|ht|hu|id|ie|il|im|in|io|iq|ir|is|it|je|jm|jo|jp|ke|kg|kh|ki|km|kn|kp|kr|kw|ky|kz|la|lb|lc|li|lk|lr|ls|lt|lu|lv|ly|ma|mc|md|me|mf|mg|mh|mk|ml|mm|mn|mo|mp|mq|mr|ms|mt|mu|mv|mw|mx|my|mz|na|nc|ne|nf|ng|ni|nl|no|np|nr|nu|nz|om|pa|pe|pf|pg|ph|pk|pl|pm|pn|pr|ps|pt|pw|py|qa|re|ro|rs|ru|rw|sa|sb|sc|sd|se|sg|sh|si|sj|sk|sl|sm|sn|so|sr|ss|st|sv|sx|sy|sz|tc|td|tf|tg|th|tj|tk|tl|tm|tn|to|tr|tt|tv|tw|tz|ua|ug|uk|us|uy|uz|va|vc|ve|vg|vi|vn|vu|wf|ws|ye|yt|za|zm|zw))?)$

Regex to pull last 2 segments from FQDN

3 Answers3