3

I'm transitioning my personal search suggestions from google to duckduckgo, but I'm missing something simple to make it work. I'm using jQuery-UI's autocomplete framework.

My search form

<form action="https://duckduckgo.com/?q=" method="post" id="search">
    <input type="text" name="query" value="" autocomplete="off">
    <button type="submit">Search</button>
</form>

My jQuery

$( "#search input[type=text]" ).autocomplete(
{
    delay: 0,
    minLength: 1,
    position: { my: "left top-3" },
    source: function( request, response )
    {
     // var suggestURL = "https://www.google.com/complete/search?client=firefox&q=%QUERY";
        var suggestURL = "https://duckduckgo.com/ac/?q=%QUERY&type=list";

        suggestURL = suggestURL.replace( "%QUERY", request.term );

        $.ajax({
            method: "GET",
            dataType: "jsonp",
            jsonpCallback: "jsonCallback",
            url: suggestURL,
            success: function( data )
            {
                response( data[1] );
            },
            error: function( jqXHR, textStatus, errorThrown )
            {
                console.log( textStatus, errorThrown );
            }
    }
});

The query for google returns:

https://suggestqueries.google.com/complete/search?client=firefox&q=foobar&callback=jsonCallback&_=1600956954436

jsonCallback && jsonCallback(["foobar",["foobar","foobar meaning","foobar google","foobar challenge","foobar2000 skins","foobar2k","foobar2000 themes","foobar2000 download","foobar2000 mac","foobar themes"],[],{"google:suggestsubtypes":[[433],[],[],[],[],[],[],[],[],[]]}])

The query for duckduckgo returns:

https://ac.duckduckgo.com/ac/?q=foobar&type=list&callback=jsonCallback&_=1600956892202

["foobar",["foobar2000","foobar","foobar2000 download","foobar ape","foobar2000 layout","foobar2000 decoder","foobar2000 tak","foobar2000 dsp"]]

The difference between the two seems to be jsonCallback && jsonCallback([data]) is included in the google query and I don't understand why they're different or how to fix it.

EDIT 1

After adding some error handling to the js, the error I'm getting is:

parsererror Error: jsonCallback was not called

EDIT 2

After digging into this a bit more, I don't think DDG's server allows it. It's my understanding that their server needs to send an appropriate response and I don't think it's doing so.

Jeff
  • 5,962
  • 16
  • 49
  • 81
  • Where do you find the duckduckgo autocomplete api docs?I'm looking for this but on their site I'm unable to find any docs about – fed3vt Sep 24 '20 at 14:46
  • Unfortunately there aren't any. Trust me. – Jeff Sep 24 '20 at 14:49
  • I've noticed it. The only way to get some info about is to inspect their search page. I was able to find this endpoint `https://duckduckgo.com/ac/?q={param}&kl=wt-wt` but I don't know for what is used the `&kl=wt-wt` param – fed3vt Sep 24 '20 at 15:04
  • 1
    The `&kl=wt-wt` is DDG's internal method of formatting the JSON results for their site. – Jeff Sep 24 '20 at 15:18
  • I'm using the endpoint without the param and it's working fine. Maybe you can do the migration by switching the used endpoint. The response in my case is in JSON and will hold eight entries with this format `{ "phrase": query }` – fed3vt Sep 24 '20 at 15:38
  • I believe DDG explicitly disallows remote calls to its' search suggestions database. Can anyone confirm? See my Edit #2 above. – Jeff Sep 24 '20 at 15:40
  • 1
    CORS problem is related to client. I've managed this error in past, you can take a look to this https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch , you need to set the `credentials` param to `include` and `mode` to `*` .If this will not work try using a proxy that will hold the request and pass back theresponse to your client app – fed3vt Sep 24 '20 at 15:43
  • Possible Duplicate: https://stackoverflow.com/questions/24377804/cross-domain-jsonp-xml-response – Twisty Sep 24 '20 at 19:19
  • See Also: http://easyautocomplete.com/example/duckduckgo-ajax-api – Twisty Sep 24 '20 at 19:21
  • @Twisty I think you're confirming my Edit #3. What I don't understand is that I'm able to see the response just fine -- it shows me the data I need, but ajax is still giving an error. Is it because of the response and/or headers sent by DDG's server? – Jeff Sep 24 '20 at 19:25
  • @Jeff a bit more seen here in regards to that: https://stackoverflow.com/questions/21885765/cross-origin-resource-sharing-cors-and-javascript – Twisty Sep 24 '20 at 19:50
  • @Twisty That's good info, thank you very much. How confident are you that CORS is the issue here? I am so bad with JS that I can't say with confidence that CORS is the problem, but I think it is. – Jeff Sep 24 '20 at 19:58
  • @Jeff very confident. – Twisty Sep 24 '20 at 20:14
  • Okay thanks. If you want to post an answer I'll mark it correct. There's a good chance I'll be trying to do this again in 2-3 years and I'll search and find this answer. Lol – Jeff Sep 24 '20 at 20:43

2 Answers2

1

Please see: https://duckduckgo.com/api

To consume it yourself, you can use one of the language libraries listed below or simply add '&format=json' (or xml if you prefer) onto any query URL in the api subdomain, e.g.

https://api.duckduckgo.com/?q=DuckDuckGo&format=json

Here are the requirements for use:

  • Attribution in each place you use our API for both us and any underlying source. For the source, you can link to the source's relevant detail page. For us, you can say Results from DuckDuckGo with our logo (and link to the specific result page).
  • Non-commercial use unless you get email approval from us (though we're generally fine with anything that isn't sketchy).
  • Use a descriptive t parameter, i.e. append &t=nameofapp to your requests.

Our overall goal is to get more people using DuckDuckGo, so please keep that in mind as well.

q: query

format: output format (json or xml)

If format=='json', you can also pass:
callback: function to callback (JSONP format)

This works successfully with JSONP: https://jsfiddle.net/Twisty/rqdtv9sn/86/

The problem here is that these are not suggestions and the URL for those, https://ac.duckduckgo.com/ac/ does not want to play nice with CORS. You can fudge around it with FETCH API yet this just proceed with the Promise even if the request fails or can't be parsed.

So until DDG offers a Suggestion API, you're mostly out of luck.

Some potential other options are discussed here: https://www.sitepoint.com/jsonp-examples/

var script = $("<script />", {
    src: "https://ac.duckduckgo.com/ac/?q=" + req.term,
    type: "application/json"
  }
);

Although that works, it doesn’t help us much, as we have no way of getting at the data it contains.

Example: https://jsfiddle.net/Twisty/rqdtv9sn/89/

The browser shows the response but then you get a Parse Error.

Twisty
  • 30,304
  • 2
  • 26
  • 45
  • Tomorrow I'm going to try out the proxy example mentioned in the excellent article you linked at sitepoint.com. Thanks for clearing this up for me. – Jeff Sep 24 '20 at 23:20
0

This is for anyone who wants to setup DuckDuckGo's search autocomplete suggestions on their server with jQueryUI's autocomplete framework.

By default you're not allowed to scrape/get data across domains with JavaScript, so I had to create a proxy file on my server to bypass these restrictions.

My hangup in this situation was that google's search autocomplete suggestions server explicitly allows these CORS violations if you pass certain variables with your request.

Here is my code to get DuckDuckGo's search autocomplete suggestions working on my web page:

index.php

<form action="https://duckduckgo.com/?q=" method="post" id="search">
    <input type="text" name="query" value="" autocomplete="off">
    <button type="submit">Search</button>
</form>

javascript.js

$( "#search input[type=text]" ).autocomplete(
{
    delay: 0,
    minLength: 1,
    position: { my: "left top-3" },
    source: function( request, response )
    {
        var suggestURL = "https://www.example.com/proxy-duckduckgo.php?q=%QUERY";

        suggestURL = suggestURL.replace( "%QUERY", request.term );
        suggestURL = suggestURL.replace( / /g, "+" )

        $.ajax({
            method: "GET",
            dataType: "json",
            url: suggestURL,
            success: function( data )
            {
                 response( data[1] );
            },
            error: function( jqXHR, textStatus, errorThrown )
            {
                console.log( textStatus, errorThrown );
            }
        });
    }
});

proxy-duckduckgo.php

<?php

$query = isset( $_GET['q'] ) ? str_replace( ' ', '+', $_GET['q'] ) : 'example';

$url = 'https://duckduckgo.com/ac/?q='.$query.'&type=list';

$ch = curl_init();
curl_setopt( $ch, CURLOPT_URL, $url );
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, TRUE );
curl_setopt( $ch, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4 );
$html = curl_exec( $ch );
curl_close( $ch );

echo $html;
Jeff
  • 5,962
  • 16
  • 49
  • 81