Parse URL to search Pattern

Question

I need to read URL content and search/regex for a pattern. e.g. in corrupt_files.jsp, I need to search for any keyword "auction_log.DATE" (where DATE is yesterday)

How can I achieve it?

Below is what I got so far:

from urllib import urlopen
import re
import time
import datetime
from datetime import date, timedelta
yesterday = date.today() - timedelta(1)

DATE= yesterday.strftime('%Y-%m-%d')

html = urlopen("http://url.com/corrupt_files.jsp").read()

for line in html.split('<tr'):
  re.search('auction_log.DATE',line)

You do know you [shouldn't parse HTML with regex](http://stackoverflow.com/a/1732454/2308683), right? — OneCricketeer, Feb 11 '16 at 15:27
And start with `'auction_log.{}'.format(DATE)` to get the correct string — OneCricketeer, Feb 11 '16 at 15:29
@cricket_007 is absolutely correct. Use an HTML parser to parse HTML. — user615501, Feb 11 '16 at 15:30

molivier · Answer 1 · 2016-02-11T15:53:29.547

0

You can use BeautifulSoup or Scrapy to extract your content. For example with BS:

from bs4 import BeautifulSoup
import urllib
r = urllib.urlopen('corrupt_files.jsp').read()
soup = BeautifulSoup(r)
soup.body.findAll(text='auction_log.DATE')

edited Feb 11 '16 at 15:53

answered Feb 11 '16 at 15:43

molivier

2,146
1
18
20

Parse URL to search Pattern

1 Answers1