Currently, I am grabbing titles using the following method:
title = html_response[/<title[^>]*>(.*?)<\/title>/,1]
This does a great job at catching "This is a title" from <title>This is a title</title>
. However, there are some web pages that open the title tag on one line, print the title on the next line, and then close the title tag.
The Ruby line I presented above doesn't catch titles such as those, so I'm just trying to find a fix for that.