How does Google+ parse URLs from posts?

Question

Google+ seems to use The-King-of-URL-Regexes to parse the suckers out of user posts. It doesn't require protocols and is good about ignoring punctuation. For example: if I post "I like plus.google.com.", the site will transform that into "I like plus.google.com." So if anyone knows of a regex that can parse URLs both with and without protocols and is good at ignoring punctuation, please answer with it.

I don't think this question is a dupe, because all the answers I've seen to similar questions seem to require a protocol in the URL.

Thanks

this blog has what you need I think. http://blog.mattheworiordan.com/post/13174566389/url-regular-expression-for-links-with-or-without-the — zer0bit, Dec 20 '12 at 14:44
@zer0bit looks like in the link you provided it couldn't match the url plus.google.com — cheesemacfly, Dec 20 '12 at 14:46
this is a tricky one... but here is a good place to start. http://mathiasbynens.be/demo/url-regex — zer0bit, Dec 20 '12 at 15:17

score 2 · Answer 1 · answered Feb 06 '13 at 07:35

Here's a more complete (full URL) implementation. Note that it is non fully RFC 3986 compliant, missing some TLDs, allows some illegal country TLDs, allows dropping the protocol part (as requested in the original Q), and has some other imperfections. The upside is that it has a lot of simplicity and is much shorter than many other implementations and does >95% of the job.

#!/usr/bin/perl -w
# URL grammar, not 100% RFC 3986 but pretty good considering the simplicity.
# For more complete implementation options see:
#   http://mathiasbynens.be/demo/url-regex
#   https://gist.github.com/dperini/729294
#   https://github.com/garycourt/uri-js (RFC 3986 compliant)
#
my $Protocol = '(?:https?|ftp)://';
# Add more new TLDs for completeness
my $TLD = '(?:com|net|info|org|gov|edu|[a-z]{2})';
my $UserAuth = '(?:[^\s:@]+:[^\s@]*@)';
my $HostName = '(?:(?:[-\w]+\.)+?' . ${TLD} . ')';
my $Port = '(?::\d+)';
my $Pathname = '/[^\s?#&]*';
my $Arg = '\w+(?:=[^\s&])*';
my $ArgList = "${Arg}(?:\&${Arg})*";
my $QueryArgs = '\?' . ${ArgList};
my $URL = qr/
    (?:${Protocol})?    # Optional, not per RFC!
    ${UserAuth}?
    ${HostName}
    ${Port}?
    (?:${Pathname})?
    (?:${QueryArgs})?
/sox;

while (<>) {
    while (/($URL)/g) {
         print "found URL: $&\n";
    }
}

score 1 · Answer 2 · answered Feb 05 '13 at 04:50

A reasonable strategy would be to use a regexp to match top level domains (TLD) preceded by a dot, and then run a known-host table lookup or DNS query as a verification step on the suspected hostname string.

e.g. here's a session using perl demonstrating the first part of the strategy:

$ cat hostname-detector
#!/usr/bin/perl -w
# Add more country/new TLDs for completeness
my $TLD = '(?:com|net|info|org|gov|edu)';
while (<>) {
    while (/((?:[-\w]+\.)+?$TLD)/g) {
         print "found hostname: $&\n";
    }
}


$ ./hostname-detector
"I like plus.google.com."
found hostname: plus.google.com

a sentence without a hostname.

here's another host: free.org
found hostname: free.org

a longer.host.name.psu.edu should work too.                    
found hostname: longer.host.name.psu.edu

a host.with-dashes.gov ...
found hostname: host.with-dashes.gov

The end goal is to hit the site and retrieve metadata, so a target verification step will happen. However, I would want all valid URLs to be detected; including forward slashes, query strings, and all the other goodies that URLs tend to contain. — JoshNaro, Feb 05 '13 at 13:49

score 0 · Answer 3 · answered Apr 18 '13 at 14:54

@arielf

It looks to me that the following line:

my $HostName = '(?:(?:[-\w]+\.)+?' . ${TLD} . ')';

should be fixed this way:

my $HostName = '(?:(?:[-\w]+\.)+' . ${TLD} . ')';

Otherwise, the input http://www.google.com gets parsed as

found URL: http://www.go
found URL: ogle.com

How does Google+ parse URLs from posts?

3 Answers3