WEB SCRAPING: Regex is not returning anything? What am I doing wrong?

Question

I am trying to write a python script which uses the "urllib" and "re" libraries to extract weather forecast information off a html page, but I cannot seem to get any values returned, could anybody help me?

import urllib
import re

url = ('http://www.metoffice.gov.uk/public/weather/forecast/gcptz5sys')

htmlfile = urllib.urlopen(url)

htmltext = htmlfile.read()

regex =('<span title="Maximum daytime temperature" data-c="10" data-f="50">(.+?)<sup>°C</sup></span>')

pattern = re.compile(regex)

temp = re.findall(pattern,htmltext)

print (temp)

I am using Python 2.7 by the way...

The MetOffice have an API for this that returns data in a parseable format; http://www.metoffice.gov.uk/datapoint — Alex K., Nov 27 '14 at 18:00
http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags — SierraOscar, Nov 27 '14 at 18:10

score 1 · Accepted Answer · answered Dec 02 '14 at 07:18

Try this:

#!/usr/bin/env python    

import urllib                                                                                                  
import re                                                                                                      


def main():                                                                                                    
    url = ('http://www.metoffice.gov.uk/public/weather/forecast/gcptz5sys')     

    htmlfile = urllib.urlopen(url)                                                                             

    htmltext = htmlfile.read()                                                                                 

    htmltext = str(htmltext).replace('\n', '')                                                                 
    htmltext = str(htmltext).replace('\t', '')                                                                 
    htmltext = str(htmltext).replace(' ', '') 

    pattern = re.compile('<spantitle="Maximumdaytimetemperature"data-c="7"data-f="45">(?P<temperature>.+?)<sup>&deg;C</sup></span>') 

    for match in pattern.finditer(htmltext):                                                                   
        print match.group('temperature')                                                                       

if __name__ == "__main__":                                                                                     
    main()

So what I did here:

Download content
Remove all new line characters
Remove all tabs
Remove all space characteres
I create and compile regex pattern where group 'temperature' will be used in order to retrieve temperature (Note: Regex dose note contains white space or new line)
Using finditer function iterate over matched elements and print it to console.

P.S.: I removed all white space charachter because it can be changed dynamically in backend and your regex should be changed every time. By remove all white space and new line characters you can avoid this problem.

WEB SCRAPING: Regex is not returning anything? What am I doing wrong?

1 Answers1