Python command prompt - automated extract links

Question

I found a good tool on github that lets you enter a URL to extract links from them: https://github.com/devharsh/Links-Extractor

However, I wanted to extract all URLs on the page, not just clickable links, for example, in the site's HTML:

<a href="www.example.com">test</a>
in plaintext HTML: www.example.com
and <img src="www.example.com/picture.png">

would print out:

www.example.com
www.example.com
www.example.com/picture.png

I'm new to python, and I haven't find any online tools that lets you extract URLs from multiple pages (I want it so you enter multiple URLs, and running it will extract all URLs from each URL you've entered), they only allow entering a single URL and extracts links from that page (one at a time).

It only prints out <a href=""></a> HTML tag urls, but not all of them.

Here is the python code (edited to handle UTF-8 and percent encoding):

#!/usr/bin/python

__author__ = "Devharsh Trivedi"
__copyright__ = "Copyright 2018, Devharsh Trivedi"
__license__ = "GPL"
__version__ = "1.4"
__maintainer__ = "Devharsh Trivedi"
__email__ = "devharsh@live.in"
__status__ = "Production"

import sys
import requests
from bs4 import BeautifulSoup
from urllib.parse import urlparse

try:

    for link in sys.argv[1:]:
        page = requests.get(link)
        soup = BeautifulSoup(page.text, "lxml")
        extlist = set()
        intlist = set()
        
        for a in soup.findAll("a", attrs={"href":True}):
            if len(a['href'].strip()) > 1 and a['href'][0] != '#' and 'javascript:' not in a['href'].strip() and 'mailto:' not in a['href'].strip() and 'tel:' not in a['href'].strip():
                if 'http' in a['href'].strip() or 'https' in a['href'].strip():
                    if urlparse(link).netloc.lower() in urlparse(a['href'].strip()).netloc.lower():
                        intlist.add(a['href'])
                    else:
                        extlist.add(a['href'])
                else:
                    intlist.add(a['href'])
        
        print('\n')
        print(link)
        print('---------------------')
        print('\n')
        print(str(len(intlist)) + ' internal links found:')
        print('\n')
        for il in intlist:
            print(il.encode("utf-8"))
        print('\n')
        print(str(len(extlist)) + ' external links found:')
        print('\n')
        for el in extlist:
            print(el.encode("utf-8"))
        print('\n')
        
except Exception as e:
    print(e)

Answer 1

Here's a quick regex to identify a URL:

(http|ftp|https)://([\w_-]+(?:(?:\.[\w_-]+)+))([\w.,@?^=%&:/~+#-]*[\w@?^=%&/~+#-])?

In practice, this would look like:

import re
import requests
import sys

def find_urls(links):
  url_list = []
  for link in links:
    page = requests.get(link).text
    parts = re.findall('(http|ftp|https)://([\w_-]+(?:(?:\.[\w_-]+)+))([\w.,@?^=%&:/~+#-]*[\w@?^=%&/~+#-])?', page)
    true_url = [p + '://' + d + sd for p, d, sd in parts]
    url_list.extend(true_url)
  return url_list

print(find_urls(sys.argv[1:]))

The output for:

print(find_urls(['https://www.google.com']))

is:

['http://schema.org/WebPage', 'https://www.google.com/imghp?hl=en&tab=wi', 'https://maps.google.com/maps?hl=en&tab=wl', 'https://play.google.com/?hl=en&tab=w8', 'https://www.youtube.com/?gl=US&tab=w1', 'https://news.google.com/nwshp?hl=en&tab=wn', 'https://mail.google.com/mail/?tab=wm', 'https://drive.google.com/?tab=wo', 'https://www.google.com/intl/en/about/products?tab=wh', 'http://www.google.com/history/optout?hl=en', 'https://accounts.google.com/ServiceLogin?hl=en&passive=true&continue=https://www.google.com/']

Thanks to Rajeev here for the regex

Edit: Given the author's updated use case, I did some trial and error and found this new regex:

((https?:\/\/.+)?(\/.*)+)

Here it is in practice:

def find_urls(links):
  url_list = []
  for link in links:
    page = requests.get(link).text
    parts = re.findall('((https?:\/\/.+)?(\/.*)+)', page)
    url_list.extend(parts)
  return url_list

I won't guarantee this will work for every use case (I'm no regex expert), but it should work for most URLs/file paths you'll find in web pages