grep multiline pattern

Question

I have a list of URLs in a file (each line = different Domain) I want to scan (not recursively) and pick two patterns, which are in different lines. After two days of trying - my head is spinning …

That is the important HTML-Part:

<a href="http://subdomain.domain.tld/">Home</a>
</li>
<li>
<a data-uv-trigger='true' href='mailto:john@doe.com'>

I need to pick the domain (subdomain.domain.tld) and the email-adress (john@doe.com). I can (wget / sed) the parts in two steps.

wget -O - -i urls-to-scan-manuell.txt | sed -n "s/\(.*a href=\"\)\(.*\)\(\">Home.*\)/\2/p"

wget -O - -i urls-to-scan-manuell.txt | sed -n "s/\(.*true' href='mailto\)\(.*\)\('>.*\)/\2/p"

But I would like to pick both parts at once and write them out to a file in one line, separated by a blank (space). It is the multiline thing with sed that drives me nuts.

Please: I need your help, would you :)

Thank you in advance, Rainer.

Answer 1

For the record: it's not recommended to parse HTML using regex.

---

You can use sed with multiple expressions with -e that could help here:

wget -O - -i urls-to-scan-manuell.txt | sed -n \
  -e "s/\(.*a href=\"\)\(.*\)\(\">Home.*\)/\2/p" \
  -e "s/\(.*true' href='mailto\)\(.*\)\('>.*\)/\2/p"

This will produce two lines, one for the domain and one for the email. If you prefer the output on one line, you can pipe to paste - -. The default delimiter is TAB, you can change that with the -d flag, so for example:

wget -O - -i urls-to-scan-manuell.txt | sed -n \
  -e "s/\(.*a href=\"\)\(.*\)\(\">Home.*\)/\2/p" \
  -e "s/\(.*true' href='mailto:\)\(.*\)\('>.*\)/\2/p" | \
paste -d, - -

Will produce:

http://subdomain.domain.tld/,john@doe.com

I took the liberty and added a : after mailto in the pattern, because I guess that was your intention.