Parse Error when making request behind proxy

Question

Hello developer community,

I'm using AWS Cloudsearch (API version: 2011-02-01) and trying to upload and search from an app living on heroku, which means i need a proxy with a static ip so that i can whitelist it, but i'm having problems...

I'm using a Heroku add-on called Proximo. Here's Heroku's code example for setting it up with RestClient:

require "rest-client"

RestClient.proxy = ENV["PROXIMO_URL"] if ENV["PROXIMO_URL"]

res = RestClient.get("http://api.someservice.com/endpoint")

puts "status code", res.code
puts "headers", res.headers

I'm using the aws_cloud_search gem which uses Faraday to make requests, and im struggling to get a proper response from CloudSearch with my patch

 # aws_cloud_search.rb

 def self.create_connection(url, aws_access_key_id=nil, aws_secret_access_key=nil)

    options = ENV['PROXIMO_URL'] ? { proxy: ENV['PROXIMO_URL'] } : {}

    connection = Faraday.new url, options do |builder|
      builder.use AWSCloudSearch::HttpCodeResponseMiddleware
      builder.use FaradayMiddleware::EncodeJson
      builder.use FaradayMiddleware::ParseJson
      builder.adapter Faraday.default_adapter  
      builder.proxy ENV['PROXIMO_URL'] if ENV['PROXIMO_URL']
    end
    connection.headers['User-Agent'] = "AWSCloudSearch-Ruby-Client/#{VERSION}"
    connection
  end

I've tried many combinations of patch work seen here (among other places) to no avail. When i set up a proxy with the ip address provided by Proximo with port 80 (as they suggest), I get the following:

Faraday::Error::ParsingError: 757: unexpected token at 'invalid'
from /app/vendor/bundle/ruby/2.0.0/gems/json-1.6.8/lib/json/common.rb:149:in `parse'
from /app/vendor/bundle/ruby/2.0.0/gems/json-1.6.8/lib/json/common.rb:149:in `parse'
from /app/vendor/bundle/ruby/2.0.0/gems/faraday_middleware-0.9.1/lib/faraday_middleware/response/parse_json.rb:11:in `block in <class:ParseJson>'
from /app/vendor/bundle/ruby/2.0.0/gems/faraday_middleware-0.9.1/lib/faraday_middleware/response_middleware.rb:48:in `call'
from /app/vendor/bundle/ruby/2.0.0/gems/faraday_middleware-0.9.1/lib/faraday_middleware/response_middleware.rb:48:in `parse'
from /app/vendor/bundle/ruby/2.0.0/gems/faraday_middleware-0.9.1/lib/faraday_middleware/response_middleware.rb:39:in `process_response'
from /app/vendor/bundle/ruby/2.0.0/gems/faraday_middleware-0.9.1/lib/faraday_middleware/response_middleware.rb:32:in `block in call'
from /app/vendor/bundle/ruby/2.0.0/gems/faraday-0.8.1/lib/faraday/response.rb:63:in `on_complete'
from /app/vendor/bundle/ruby/2.0.0/gems/faraday_middleware-0.9.1/lib/faraday_middleware/response_middleware.rb:30:in `call'
from /app/vendor/bundle/ruby/2.0.0/gems/faraday_middleware-0.9.1/lib/faraday_middleware/request/encode_json.rb:23:in `call'
from /app/vendor/bundle/ruby/2.0.0/gems/faraday-0.8.1/lib/faraday/response.rb:8:in `call'
from /app/vendor/bundle/ruby/2.0.0/gems/faraday-0.8.1/lib/faraday/connection.rb:226:in `run_request'
from /app/vendor/bundle/ruby/2.0.0/gems/faraday-0.8.1/lib/faraday/connection.rb:99:in `post'
from /app/vendor/bundle/ruby/2.0.0/bundler/gems/aws_cloud_search-0d5b94169466/lib/aws_cloud_search/cloud_search.rb:18:in `documents_batch'
from (irb):9
from /app/vendor/bundle/ruby/2.0.0/gems/railties-3.2.19/lib/rails/commands/console.rb:47:in `start'
from /app/vendor/bundle/ruby/2.0.0/gems/railties-3.2.19/lib/rails/commands/console.rb:8:in `start'
from /app/vendor/bundle/ruby/2.0.0/gems/railties-3.2.19/lib/rails/commands.rb:41:in `<top (required)>'
from script/rails:6:in `require'
from script/rails:6:in `<main>'irb(main):010:0> ENV['PROXIMO_URL']

=> "http://nn.nnn.nn.nn:80" (i've obscured my ip address)

I take this to mean that the actual key being parsed is 'invalid', and i get this on search and document (i'm not using ssl). (i get html with 403 Forbidden without the proxy). I've attempted to print out the response body but I think the error is occurring before I can even print it out (as with the 403).

Anything obvious that I haven't tried or problems with my approach?

Thanks

I don't know anything about Faraday, but it the "invalid" token the actual proxy URI? It seems like Faraday is trying to parse it and fails? Also, it looks like you're trying to set the proxy twice: once in `options` and another in an explicit call. I say get rid of both, and set the env variable `http_proxy`. Faraday uses it. — Nitzan Shaked, Jul 21 '14 at 23:41

score 0 · Accepted Answer · answered Jul 22 '14 at 02:17

0

The aws-sdk-core gem now supports the CloudSearch domain APIs (Search, Suggest, UploadDocuments). You should be able to install the gem and configure the client with a proxy:

require 'aws-sdk-core'

client = Aws::CloudSearchDomain::Client.new(
  endpoint:'http://...',
  http_proxy: 'http://...')

The client API reference can be found here: http://docs.aws.amazon.com/sdkforruby/api/Aws/CloudSearchDomain/Client.html

answered Jul 22 '14 at 02:17

Trevor Rowe

6,499
2
27
35

Thanks for your input. I actually ended up using RestClient cuz it was nice and simple, and not a dev release. My initial problem was that i didn't read the docs properly on Proximo and actually set the PROXIMO_URL myself, setting it to http:// instead of the more complex one with a username and password. Even with that problem solved, Faraday just wouldn't have it, so instead of patching the aws_cloudsearch_gem, i just used RestClient for batch uploading. @Nitzan Shaked, thank u for pointing out the http_proxy var. Unfortunately, i wasn't able to get that to work either. – Jeremy Eaton Jul 30 '14 at 21:13

Parse Error when making request behind proxy

1 Answers1