22

I am making a series of API calls using httparty. The first two API calls succeed, but the third one fails. It pauses for about 60 seconds (default timeout period) and then returns this error:

/Users/luigi/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/protocol.rb:158:in `rescue in rbuf_fill': Net::ReadTimeout (Net::ReadTimeout)
    from /Users/luigi/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/protocol.rb:152:in `rbuf_fill'
    from /Users/luigi/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/protocol.rb:134:in `readuntil'
    from /Users/luigi/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/protocol.rb:144:in `readline'
    from /Users/luigi/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http/response.rb:39:in `read_status_line'
    from /Users/luigi/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http/response.rb:28:in `read_new'
    from /Users/luigi/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1406:in `block in transport_request'
    from /Users/luigi/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1403:in `catch'
    from /Users/luigi/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1403:in `transport_request'
    from /Users/luigi/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1376:in `request'
    from /Users/luigi/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1369:in `block in request'
    from /Users/luigi/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:852:in `start'
    from /Users/luigi/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1367:in `request'
    from /Users/luigi/.rvm/gems/ruby-2.0.0-p247@pdf/gems/httparty-0.12.0/lib/httparty/request.rb:93:in `perform'
    from /Users/luigi/.rvm/gems/ruby-2.0.0-p247@pdf/gems/httparty-0.12.0/lib/httparty.rb:486:in `perform_request'
    from /Users/luigi/.rvm/gems/ruby-2.0.0-p247@pdf/gems/httparty-0.12.0/lib/httparty.rb:423:in `get'
    from /Users/jmccann/.rvm/gems/ruby-2.0.0-p247@pdf/gems/httparty-0.12.0/lib/httparty.rb:518:in `get'

My question is why is this happening? Is this error indicative of an error with the API, or is there something I could do to cause it?

My code:

This is the call that works:

url = HTTParty.get("https://dev.test.com#{call}",
    :basic_auth => auth,
    :headers => {'Accept' => 'application/json' } )

This is the call that doesn't work:

url = HTTParty.get("https://dev.test.com#{call}",
    :basic_auth => auth,
    :headers => {'Accept' => 'application/json' } )

The only thing that changes at all is the actual call, but they are both valid calls according to the API Documentation.

Community
  • 1
  • 1
Luigi
  • 5,443
  • 15
  • 54
  • 108
  • 1
    Yeah, my team has been getting it too: /home/travis/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/net/protocol.rb:158:in `rescue in rbuf_fill': Net::ReadTimeout (Net::ReadTimeout) – RudyOnRails Jan 29 '14 at 16:46
  • Did you ever get this figured out? I'm having this issue too. – Joshua F. Rountree Feb 14 '14 at 13:34
  • I did not - Unfortunately, I don't even recall where exactly I was having this issue to try and replicate it. I ended up switching to the `typhoeus` gem, and it has worked fine with my application. Let me know if you need any help switching to `typhoeus` and I'll be glad to write something out for you. – Luigi Feb 14 '14 at 19:45
  • 1
    Perhaps rate-limiting on the API side? This might be the case if you are calling an API that limits the # of calls in a second, for instance. – jschorr Mar 28 '14 at 21:01
  • You could use tcpdump -i piped through grep on the IP address you are trying to communicate with to see if it exposes any issues. Probably not since you are using SSL/HTTPS. You could look at using some sort of proxy like charles to sit between – Lance Woodson Apr 19 '14 at 01:37
  • @jschorr, I second this: maybe the API server can respond to a limited (1?) number of simultaneous requests. @Luigi, try to call this same URL with the `curl` command. – Cédric Darné May 21 '14 at 08:39
  • @Luigi - How did you handle this? I am having the same problem – Sam Oct 08 '14 at 07:19
  • both blocks of code are the same ??? – Michael Durrant Oct 20 '16 at 21:05

1 Answers1

15

From the Ruby docs: http://ruby-doc.org/stdlib-2.1.2/libdoc/net/http/rdoc/Net/HTTP.html

Number of seconds to wait for one block to be read (via one read(2) call). Any number may be used, including Floats for fractional seconds. If the HTTP object cannot read data in this many seconds, it raises a Net::ReadTimeout exception. The default value is 60 seconds.

What is call? Are the service endpoints different? Are you paging through records? APIs often rate limit callers to prevent DDOS attacks. You can try wrapping your api calls in some retry logic or adding some sleep code.

AL.
  • 36,815
  • 10
  • 142
  • 281
jaysqrd
  • 375
  • 4
  • 13
  • @Luigi - How did you handled this? I am having the same problem – Sam Oct 08 '14 at 07:18
  • I was hitting this error form sidekiq workers hitting a third party API. If your queueing the work, I'd checkout Sidekiq's Delay Extensions: https://github.com/mperham/sidekiq/wiki/Delayed-extensions – DrMavenRebe Oct 27 '14 at 23:54
  • @Luigi I've used this gem: https://github.com/ooyala/retries to handle retries. You can control how many retries you want to allow as well as the exceptions that it will rescue. – jaysqrd Dec 05 '14 at 07:11
  • @jaysqrd or any one let me know, how we can upload huge no.of records to API as post? around 1 lac records, ideally we couldn't send the within single API call. but we can send limit collection in multiple calls. Is it allow API for multiple calls. – Raju akula Apr 15 '15 at 12:31
  • @Rajuakula sorry for the late reply. Does the API support batching up your data? – jaysqrd Sep 16 '15 at 18:26