0

I'm trying to do some basic web scraping and eventually found the need to limit my request calls as the server will return a page not found when there's too many request

Currently I'm using request-promise wrapped in request-promise-retry to make this call and I also found this article which seems to be trying to achieve the same thing Throttle and queue up API requests due to per second cap

I went ahead to try the simple-rate-limiter as it looks simple enough to use but got the following error 

TypeError: requestPromise(...).catch is not a function
    at promiseRetry.retries (C:\project\node_modules\request-promise-retry\index.js:18:27)
    at C:\project\node_modules\promise-retry\index.js:29:24
    at <anonymous>

My guess is that the simple-rate-limiter doesn't work with request-promise and only works with request.

Are there any simple ways to go around throttling the requests without having to rewrite all my calls with "request" instead of "request-promise"?

Or is it a good idea to rewrite using normal "request", which will be a pain as I already have a few pieces of code written with request-promise and is expecting a promise to be returned.

Jeremy Kuah
  • 519
  • 1
  • 6
  • 18
  • What version of node are you using? Couple warning signs on the simple-rate-limiter repo: "Currently works with node.js v0.10.1+ (and probably lower)" (current stable is v6, going on v8), and commit activity is lacking after 2014. – stealththeninja Aug 28 '17 at 03:29
  • @stealththeninja I'm on v8.3.0, hmm so probably a bad idea to use simple-rate-limiter huh. – Jeremy Kuah Aug 28 '17 at 03:33

0 Answers0