Parse HTML using perl regex

Question

I created a Perl script that would use an online website to crack MD5 hashes after the user inputs the hashes. I am partially successful as I am able to get the response from the website, though I need to parse the HTML and display the hash, and corresponding password in clear text to the user. The following is the output snippet I get now:

<strong>21232f297a57a5a743894a0e4a801fc3</strong>: admin</p>

Using regex buddy, I was able to use the following expression [a-z0-9]{32} to match the hash part alone. I need the final output in the following format:

21232f297a57a5a743894a0e4a801fc3: admin

Any help would be appreciated. Thank you!

Answer 1

Ten years later, we have more sophisticated and easier solutions.

If you can write a CSS selector, you can easily pull out the parts of a response without dealing with the complexity of HTML::Parser and other approaches. Mojo::UserAgent does it all for you:

use v5.16;

# This is a real URL just for this answer
my $url = 'https://gist.githubusercontent.com/briandfoy/85033496f93e860cdf53f45ba931e8f7/raw/a0876e090fc5a3c1b75ff2a19580f731b784828d/selector_example.html';

use Mojo::UserAgent;
my $ua = Mojo::UserAgent->new;

my $tx = $ua->get($url);
my $hash = $tx->res->dom->at( 'div p strong' )->text;  # or whatever selector

say $hash;

The at only finds the first node that matches the selector. If there are multiple nodes, use find instead. You get matches as a "collection" and transform them the same way inside map:

my $hashes = $tx->res->dom->find( 'div p strong' )
    ->map( sub {$_->text} )
    ->join("\n");

The particular selector, div p strong, depends on the HTML you get back, and is much easier to pinpoint an element if there are id or class values. I go through these extensively in Mojo Web Useragents.

Answer 2

I think you'd be much better off using HTML::Parser to simply/reliably parse that HTML. Otherwise you're into the nightmare of parsing HTML with regexps, and you'll find that doesn't work reliably.

Answer 3

There are a few tools that can handle both fetching and parsing the page for you available on CPAN. One of them is Web::Scraper. Tell it what page to fetch and which nodes (in xpath or CSS syntax) you want, and it will get them for you. I'll not give an example as I don't know your URL.

There is a good blogpost about this on blogs.perl.org by stas that uses a different module that might also be helpful.

Answer 4

Here it is:

$str = q{<strong>21232f297a57a5a743894a0e4a801fc3</strong>: admin</p>};
@arr = $str =~ m{<strong>(.+)</strong>(.+)</p>};
print(join("", @arr), "\n");

Answer 5

So, doing this reliably in Perl is possible because Perl's regexps have an extended format with the complexity required to parse HMTL (you can actually embed Perl code in your regex and your regex within Perl code, but more to the point here, you can have recursive regular expressions that can actually parse HTML). However, you really don't want to do this. If you decide to do so anyway, please read up on advanced Perl regular expressions so you have some idea of what you are getting into.

The actual problem here is that parsing things like HTML reliably is an immense process and it is extremely tempting to use naive regex implementations to do this. Those will break because regex is (outside of Perl) a regular language and thus not capable of parsing anything other than a regular language. Perl is different and the addition of things like recursive regular expressions make this possible but very difficult, bug prone, and complex. Moreover, you would probably only be able to support, realistically, one variant of HTML.

HTML::Parser does the main work in a C, XS module which can take advantage of pre-existing C libraries. This means, importantly not maintaining for yourself a massive nightmarish codebase.