13

I am trying to build book title recommendation system using Google Books API. Unfortunately results I get are extremely irrelevant comparing to https://books.google.com . For example that's a list I get from search by word "sher"(looking forward to something like Sherlock Holmes primarily).

`She Said Yes;The Oh She Glows Cookbook;What Can She Know?;She-Wolf;Murder She Wrote;My Mother She Killed Me, My Father He Ate Me;22 Things a Woman Must Know If She Loves a Man with Asperger's Syndrome;Where She Danced;The Israeli-Palestinian Peace Negotiations, 1999-2001`

As you see, there is no even most relevant title. If you type "Sher" at google books website you'll get absolutely correct relevant recommendations. Have I understood Google API right? What is wrong in my code?

            var request = 'https://www.googleapis.com/books/v1/volumes';
            var outputbookslistshow = 0; 
            $('#myTextArea').on('keyup', function(){
                if(outputbookslistshow == 0){
                  outputbookslistshow = 1;
                  $('#outputgbooksrec').show(300); // it's a div for outputting titles
                }
            $('#outputgbooksrec').empty();
                 var keywords = $(this).val();
                if(keywords.length > 0 ){
                      $.getJSON(request + '?q=' + keywords, function(response){
                        for (var i = 0; i < response.items.length; i++) {
                    var item = response.items[i];
             document.getElementById("outputgbooksrec").innerHTML += "<br>"
 + "<div class='itembook'>" + item.volumeInfo.title + "</div>";
        } 
        })
        }

    });
  • 1
    I think this is relevant : `Like Google's web search, the API searches on complete words (and related words with the same stem), not substrings.` See [documentation](https://developers.google.com/books/docs/v1/using?hl=es#st_params) – Eric Martinez Aug 08 '15 at 12:16
  • Even searches with complete words are quite irrelevant: https://www.google.com/search?tbm=bks&q=harry+potter – AryanJ-NYC Oct 23 '17 at 15:51

1 Answers1

12

0. TLDR

Here's a working fiddle using Google's https://suggestqueries.google.com/complete/search

Parameters:

output/client  # "toolbar" => xml, "firefox" => json, "chrome" => jsonp
ds             #  which site to search in ("bo" for books, "yt" for youtube...)
q              #  search term: "sher"

Query:

https://suggestqueries.google.com/complete/search?output=firefox&ds=bo&q=sher

Results:

["sher",["sherlock holmes","sherrilyn kenyon","sherman alexie","sheryl sandberg","sherlock","sherlock holmes short stories","sherlock holmes book","sher o shayari","sherlock holmes novels","sher shah suri"]]

1. Suggestions vs Search Results

The first thing to realise is that when Google makes suggestions, they are not the results it would show you if you were to hit enter.

Search Results are relevant if relevant terms are included in your query.

Suggestions assume that your query is incomplete and therefore compare your query to other queries to guess what the completed version of your query might be.

When I search "sher" on http://books.google.com the results I see are:

  • The Israeli-Palestinian Peace Negotiations, 1999-2001
  • Beyond Neutrality: Perfectionism and Politics
  • Desert
  • Refuse to Choose!: Use All of Your Interests, Passions, ...

The reason for this is the author: In the case of the first three, "George Sher" and in the case of the fourth "Barbara Sher". This is desired behaviour because when I search "sher" I don't want "Sherlock" results burying "George Sher".


2. The Solution

Google has a kind of API for its suggestions as well. Some information about it can be found here. More significantly, using the developer tools however, you can see precisely what Google is doing.

Using Developer Tools: Inspect the https://books.google.com page (CTRL+SHIFT+i in Chrome). Go to the network tab and wait until everything is loaded.

When you begin typing, Google fires requests to the server which you will see populate in the list. When I typed "sher", Google sent this request:

https://suggestqueries.google.com/complete/search?client=books&ds=bo&q=sher&callback=_callbacks_._1id33zyi5

Look at the variables:

client   = books
ds       = bo
q        = sher
callback = _callbacks_._1id33zyi5
  • client determines the type of result that you receive (XML [toolbar], JSON [firefox], JSONP [chrome])
  • ds limits the search to a specific site (books [bo], youtube [yt] etc.).
  • q is, of course, the query text
  • callback is a paramater used for JSONP (which has some important differences to JSON). Don't worry too much about it because jQuery can handle this for you.

I pieced together bits of information on these parameters by looking at this request and by reading this and this.

CORS: Because you are making a request from a domain that's not google.com, you are going to get an Access-Control-Allow-Origin error. This is a security measure trying to prevent XSS. To get around this problem, you will need to use JSONP.

Using jQuery, we needn't worry about the callback so let's change the client parameter to chrome and use this final query:

https://suggestqueries.google.com/complete/search?client=chrome&ds=bo&q=sher

Working Example Below: In this example, you may want to take note of the "google:suggestrelevance" key which is an added bonus of using JSONP (Google only returns that information in JSONP data).

var requestUrl = "https://suggestqueries.google.com/complete/search?client=chrome&ds=bo&q=";
var xhr;

$(document).on("input", "#query", function () {
    typewatch(function () {
        // Here's the bit that matters
        var queryTerm = $("#query").val();
        $("#indicator").show();
        if (xhr != null) xhr.abort();
        xhr = $.ajax({
            url: requestUrl + queryTerm,
            dataType: "jsonp",
            success: function (response) {
                $("#indicator").hide();
                $("#response").html(syntaxHighlight(response));
            }
        });
    }, 500);
});


/*
 *  --------- YOU ONLY NEED WHAT IS ABOVE THIS LINE ---------
 */
$(document).ready(function () {
    $("#indicator").hide();
});

// Just for fun, some syntax highlighting...
// Credit: http://stackoverflow.com/a/7220510/123415
function syntaxHighlight(json) {
    if (typeof json != 'string') {
        json = JSON.stringify(json, undefined, 2);
    }
    json = json.replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;');
    return json.replace(/("(\\u[a-zA-Z0-9]{4}|\\[^u]|[^\\"])*"(\s*:)?|\b(true|false|null)\b|-?\d+(?:\.\d*)?(?:[eE][+\-]?\d+)?)/g, function (match) {
        var cls = 'number';
        if (/^"/.test(match)) {
            if (/:$/.test(match)) {
                cls = 'key';
            } else {
                cls = 'string';
            }
        } else if (/true|false/.test(match)) {
            cls = 'boolean';
        } else if (/null/.test(match)) {
            cls = 'null';
        }
        return '<span class="' + cls + '">' + match + '</span>';
    });
}

// And automatic searching (when you stop typing)
// Credit: http://stackoverflow.com/a/2219966/123415
var typewatch = (function () {
    var timer = 0;
    return function (callback, ms) {
        clearTimeout(timer);
        timer = setTimeout(callback, ms);
    };
})();
/* 
 * Safe to ignore:
 * This is just to make stuff look vaguely decent
 */
body {
  padding: 10px;
}
div * {
    vertical-align: top;
}
#indicator {
    display: inline-block;
    background: no-repeat center/100% url('http://galafrica.actstudio.ro/img/busy_indicator.gif');
    width: 17px;
    height: 17px;
    margin: 3px;
}
/*
 *
 * CREDIT:
 * http://stackoverflow.com/a/7220510/123415
 */
 pre {
    outline: 1px solid #ccc;
    padding: 5px;
}
.string {
    color: green;
}
.number {
    color: darkorange;
}
.boolean {
    color: blue;
}
.null {
    color: red;
}
.key {
    color: #008;
}
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<div>
    <input type=text id="query" placeholder="Start typing..." /><span id="indicator"></span>

</div>
<pre id="response"></pre>
jcuenod
  • 55,835
  • 14
  • 65
  • 102
  • XMLHttpRequest cannot load https://suggestqueries.google.com/complete/search?output=firefox&ds=bo&q=He. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:8888' is therefore not allowed access. –  Aug 10 '15 at 11:53
  • @Vinand I have added a fiddle and updated (to use JSONP), let me know if you're still having issues... – jcuenod Aug 11 '15 at 17:59
  • It works great, but is there any way to get additional information on every title? Like if we use usual search we can get back isbn, coverurl, author and other volumeInfo? Or the only way is to search for a chosen suggestions? –  Aug 12 '15 at 14:09
  • And if search of definite suggestion is the only way, how to actually select this suggestion? –  Aug 12 '15 at 15:03
  • 2
    @Vinand it sounds as though you are asking about the interface now which is well beyond the scope of this question. You will need to make other requests though (you could even use the google api to search based on the suggestions that you get from my code). How you display and present that is another question. – jcuenod Aug 13 '15 at 11:36
  • @Mousey http://stackoverflow.com/questions/32011088/how-to-represent-and-autocomplete-json-of-suggestions-based-on-google-books-api –  Aug 14 '15 at 13:20
  • @jcuenod http://stackoverflow.com/questions/32011088/how-to-represent-and-autocomplete-json-of-suggestions-based-on-google-books-api –  Aug 14 '15 at 13:20
  • @jcuenod but Google do not actually provide any official API and documentation on suggestions? However, it's very important functionality for developers –  Aug 14 '15 at 13:47
  • 1
    @Vinand I can only speculate but maybe Google is not interested in helping people provide suggestions - if you want to use google search, use google search. Google is providing an api for actual data when you have a product in mind. – jcuenod Aug 14 '15 at 22:53