Regulating / rate limiting ruby mechanize

Question

I need to regulate how often a Mechanize instance connects with an API (once every 2 seconds, so limit connections to that or more)

So this:

instance.pre_connect_hooks << Proc.new { sleep 2 }

I had thought this would work, and it sort of does BUT now every method in that class sleeps for 2 seconds, as if the mechanize instance is touched and told to hold 2 seconds. I'm going to try a post connect hook, but it is obvious I need something a bit more elaborate, but what I don't know what at this point.

Code is more explanation so if you are interested following along: https://github.com/blueblank/reddit_modbot, otherwise my question concerns how to efficiently and effectively rate limit a Mechanize instance to within a specific time frame specified by an API (where overstepping that limit results in dropped requests and bans). Also, I'm guessing I need to better integrate a mechanize instance to my class as well, any pointers on that appreciated as well.

http://www.ruby-doc.org/core-1.9.2/Kernel.html#method-i-sleep I'm guessing sleep is too big a hammer for this — blueblank, Feb 11 '12 at 19:08

score 2 · Accepted Answer · answered Feb 12 '12 at 06:50

2

Pre and post connect hooks are called on every connect, so if there is some redirection it could trigger many times for one request. Try history_added which only gets called once:

instance.history_added = Proc.new {sleep 2}

answered Feb 12 '12 at 06:50

pguardiario

53,827
19
119
159

This seems to be working, and not making a mess of everything else so far. – blueblank Feb 12 '12 at 20:12

score 1 · Answer 2 · answered May 09 '12 at 05:41

1

I use SlowWeb to rate limit calls to a specific URL.

require 'slowweb'
SlowWeb.limit('example.com', 10, 60)

In this case calls to example.com domain are limited to 10 requests every 60 seconds.

answered May 09 '12 at 05:41

Daniel Da Cunha

1,004
11
15

Regulating / rate limiting ruby mechanize

2 Answers2

Linked