How to retrieve the HTML of a page from CommonCrawl?

Question

Assuming I have:

• the link of the CC*.warc file (and the file itself, if it helps);
• offset; and
• length

How can I get the HTML content of that page?

Thanks for your time and attention.

Answer 1

Using warcio it would be simply:

warcio extract --payload <file.warc.gz> <offset>

Alternatively, fetch the WARC record using the HTTP range request and then extract the payload at offset 0:

curl -s -r331727487-$((331727487+6613-1)) \
   https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2020-40/segments/1600400203096.42/warc/CC-MAIN-20200922031902-20200922061902-00310.warc.gz \
   >warc_temp.warc.gz
warcio extract --payload warc_temp.warc.gz 0

The range starts at offset and ends at offset+length-1. See also getting WARC file