Scrape multiple pages of DuckDuckGo's HTML

Asked Jun 18 '21 at 00:52

Active Jun 18 '21 at 14:34

Viewed 306 times

I've created a simple PHP script to scrape DuckDuckGo's html.duckduckgo.com. I'm looking to scrape more than the first page of results. The library I am using only supports GET requests and I can't seem to find the GET param that will show a new page of results.

Pressing the "next" button sends a POST request to DDG, and I've tried adding params from the POST request to the GET request with no luck.

Any idea how I could find the GET param to see the next page of results? Does such a thing even exist?

This is my current code, I'm using DiDOM.

<?php

require "../vendor/autoload.php";

use DiDom\Document;

$document = new Document('https://html.duckduckgo.com/html/?q=test', true);

$posts = $document->find('.result__url');

$links = [];

foreach($posts as $post) {
    array_push($links, trim($post->text()));
}

$api = json_encode($links);
file_put_contents("links.json", $api);

edited Jun 18 '21 at 14:34

asked Jun 18 '21 at 00:52

x43

Is there code to go with this? Should it have a php tag? – QHarr Jun 18 '21 at 05:21
Its not much of a PHP question, its more of how I would go about scraping multiple pages of DDG – x43 Jun 18 '21 at 14:29
Scraping how? Are you not using php to perform this scraping? Labelling with the intended language / framework will better draw the attention of the relevant experts. – QHarr Jun 18 '21 at 14:30
Fair, I edited my post. – x43 Jun 18 '21 at 14:34
you can use `cURL` for fetching data pass it to `DiDom` – gaurav Jun 18 '21 at 14:53

Scrape multiple pages of DuckDuckGo's HTML

0 Answers0