0

I'm using an API - 'Have I been pwned?' which is rate limited - "limited to one per every 1500 milliseconds". I have looked at quite a few other questions on here, researched via google and a couple of other forum sites as well as tried myself to find a solution to this one.

Does the Javascript function

setInterval()

Really work for this kind of issue or problem? Has anyone found a solution that effectively works? I'm kinda at my wit's end with this one as

var url = "https://haveibeenpwned.com/api/v2/breachedaccount/";
var breach = Array();
setInterval($.ajax({
           url: url,
           type: 'GET',
           dataType: 'JSON',
           success: function(data) {
                                       breach[] = data;
                    }), 15000);

Does not seem to work, especially where my current project is storing the information for multiple email addresses. So for example if I store 4 email addresses in an array and want to loop through but wait the 1500 ms before hitting the API again to query for the next email address.

Any ideas anyone? or is there a NodeJS solution to this that might work as I've been learning that recently too.

Thank you in advance

Dave
  • 45
  • 7
  • Note that setInterval expects function as first argument. `$.ajax` will get called immediately and returns a promise object not function. Wrapping it in `function(){ /* $.ajax here*/}` would make it do what you were expecting – charlietfl Sep 02 '18 at 21:06
  • @charlietfl What about in the case of several email addresses? That is part of the project I'm working on but if it calls the AJAX request straight away without waiting at least the 1500 ms, it will hit the API rate limit. I can't seem to figure out a way of doing it. It would have to go through a `for(){ .... }` loop to get each email address to say I limit to just 4 emails to be checked and they're stored in `var emails = [];` I just don't see how to work around it. – Dave Sep 02 '18 at 22:36
  • In a loop can use setTimeout instead of setInterval and multiply `1500` times index of loop. First would be `0*1500` ... then `1*1500`, `2*1500` etc. Assumes you only have to do this once and aren't doing this all the time in app otherwise need a queue – charlietfl Sep 02 '18 at 22:41
  • @charlietfl I tried doing something similar but it didn't seem to work. I've tried various times to see if it will work but it just keeps hitting the rate limit or code 429 as it is called in the API. Here is a JS Fiddle I've created: https://jsfiddle.net/xpvt214o/723345/ – Dave Sep 02 '18 at 23:10
  • your function doesn't return anything and goes into an infinite loop. Also note that a return in a callback doesn't return to outer function and `$.ajx` is asynchronous – charlietfl Sep 02 '18 at 23:27
  • This is more what I was suggesting https://jsfiddle.net/d4cs1nef/6/. If you get a 429 then set another timeout inside the statuscode callback to make the request again but keep track of how many attempts so it doesn't go into infinite loop also – charlietfl Sep 02 '18 at 23:28
  • That is what I'm trying to work out. I have it at least returning the Title of a breach the email address has been in, however I do not know where to set the `clearTimeout(timerId);` – Dave Sep 02 '18 at 23:29
  • Shouldn't need to clear it. SetTimeout is a one time thing – charlietfl Sep 02 '18 at 23:30
  • wierd api to only allow one request every 1.5 seconds. Can understand x requests per minute but never seen one throttled this way – charlietfl Sep 02 '18 at 23:33
  • @charlietfl I've just run a test on it and it's only getting the first email address and not outputting any results. I've even tried `console.log(email)` or `console.log(emails[i])` and all I get is `undefined` – Dave Sep 02 '18 at 23:43
  • My bad. Forgot to unwind the `for loop` Change `var i=0` to `let i=0` OR use `emails.forEach closure`. https://jsfiddle.net/d4cs1nef/9/ See [JavaScript closure inside loops – simple practical example](https://stackoverflow.com/questions/750486/javascript-closure-inside-loops-simple-practical-example) – charlietfl Sep 02 '18 at 23:47
  • @charlietfl Thank you so much! That works an absolute treat. I will credit your username in my code as thanks for helping me out! – Dave Sep 02 '18 at 23:54
  • @charlietfl Oh, sorry man one other thing forgot to ask. How do I know or how can I attach something to let me know the timer is done or the for loop is done? Is there another callback I can attach? Thanks. – Dave Sep 03 '18 at 00:39
  • Bit more elaborate approach using `Promise.all()`. https://jsfiddle.net/d4cs1nef/28/ – charlietfl Sep 03 '18 at 01:14
  • @charlietfl That's actually really cool, unfortunately if it hits a 404 status code which is returned if the email address isn't found, it halts the promise and doesn't output the various breach information. As seen here: https://jsfiddle.net/drr10/n7b3okyx/ – Dave Sep 03 '18 at 01:35
  • Ok getting more complex but you can add an intermediate `catch` but have to return something from that catch to allow the rest of the promise chaining to keep working...then filter the responses at the end to remove what you returned – charlietfl Sep 03 '18 at 01:38
  • @charlietfl Trying to put the information into a `breaches[]` array but the console is telling me that `cannot read property 'push' of undefined. I've updated the jsfiddle - https://jsfiddle.net/drr10/n7b3okyx/2/ – Dave Sep 03 '18 at 01:57
  • You have a misunderstanding of array syntax in javascript. Noticed it in your original post also. Looks like you are trying to use php syntax. `breaches['title']` isn't an array and an array in javascript only has numeric indexes. No associative arrays in javascript – charlietfl Sep 03 '18 at 02:02
  • @charlietfl Hmm yeah I see your point. Still though I think the ultimate goal is to push each lot of breach information into an array if it possible then return the array and use it after the promise has completed. Also I've been tinkering with the code again and still can't see where I can catch any 404 - email not found and then out put that as `$('#results').append('

    Email not found!

    ');` for each one.     – Dave Sep 03 '18 at 02:16

1 Answers1

0

I would create a queue with requests and process them one by one.

function myRequestQueue()
{
    this.queue = [];
    this.timer = null;
    this.add = function(params)
    {
       this.queue.push(params);
    }

    this.processQueue = function()
    {
         if (this.queue.length > 0)
         {
              $.ajax(this.queue.shift());   
         }

    }
    this.timer = setInterval(this.proceesQueue,1500)

}

Usage:

 var myQ = new myRequestQueue(), breach = [];

 for(var i=0;i<100;i++){

     myQ.add({
       url: url,
       dataType: 'JSON',
       success: function(data) { breach.push(data) }});
 } // 100 requests

 // expect them to be processed one by one every 1500 ms

You may want to add a callback when the queue is emptied or something depending on your exact use case

Eriks Klotins
  • 4,042
  • 1
  • 12
  • 26
  • I tried adding `console.log(data)` and it doesn't seem to be doing anything at all. Is there something wrong with it? That isn't something to be used in NodeJS is it? If so I'm only using it in a general web page using JQuery. – Dave Sep 02 '18 at 21:27
  • You are mixing things up.. jQuery is meant for frontend, NodeJS works on backend. Where do you want your code to run? Besides, from a frontend you cannot call 3rd party urls. So, please give some more details and context on what you are trying to do – Eriks Klotins Sep 02 '18 at 21:31
  • In answer to your question at the moment I'm building a basic web page that will take no more than 4 emails at a time, these are fed into an array e.g. `var emails = [];` a for loop goes through each email and queries HIBP API if the email address found the breaches for that address are fed into `var breach = []` array. As said in my original question the API is rate limited to one query per 1500 ms. I'm more familiar with JQuery so that is why I asked if there is a solution using that but was also curious as to whether there was a NodeJS solution also. Hopefully that will answer the question. – Dave Sep 02 '18 at 21:39
  • Tried putting it into a `index.js` file and running through node but it gave me this error: `TypeError [ERR_INVALID_CALLBACK]: Callback must be a function` – Dave Sep 02 '18 at 22:12
  • Any ideas here @Eriks Klotins ?? – Dave Sep 02 '18 at 22:37
  • Well, which exact line generates the error? I suspect wrong argument order.. Besides, on NodeJS use http package (https://nodejs.org/api/http.html) to make http requests. – Eriks Klotins Sep 03 '18 at 06:58