HTTPoison manually close connection

Question

This is a very simple module that keeps checking the requester's IP. I use backconnect proxies which means that it gets new IP on every http request.

defmodule Test do
  def go() do
    Enum.each(1..1, fn x ->
    Task.Supervisor.async_nolink(Scraper.TaskSupervisor, fn ->
      r = HTTPoison.get("https://api.ipify.org?format=json", [],
        [timeout: 10_000, recv_timeout: 10_000, proxy: "ip:port", ssl: [{:versions, [:'tlsv1.2']}]])

      case r do

        {:ok, %HTTPoison.Response{status_code: 200, body: body}} ->
          IO.inspect body |> Jason.decode
          :timer.sleep(1000)
          go()
      end
    end)
    end)

  end
end
Test.go()
:timer.sleep(2000000)

Problem? HTTPoison(hackney) doesn't release the connection as long as process is alive so IP is always the same. How would I manually close the connection inside:

{:ok, %HTTPoison.Response{status_code: 200, body: body}} ->
          IO.inspect body |> Jason.decode

score 2 · Answer 1 · answered Mar 18 '19 at 06:17

From the HTTPoison README:

Normally hackney opens and closes connections on demand, but it also creates a default pool of connections which are reused for requests to the same host. If the connection and host support keepalive, the connection is kept open until explicitly closed.

HTTPoison does not export the close for the underlying socket, and there is an open issue for that since 2017.

Meanwhile, you might call hackney_pool:stop_pool(:default). This is obviously not the optimal solution by any mean and it has a huge overhead. So my advise would be to either provide a pull request to HTTPoison enabling close connection functionality (via delegating to ;hackney_response.close/1) or simply get rid of redundant HTTPoison and just use hackney.

score 2 · Answer 2 · edited Mar 19 '19 at 00:48

2

Simplest solution turned out bo be setting pool: false, the entire request looks like:

HTTPoison.get(url, [], [
  timeout: 5_000,
  recv_timeout: 5_000,
  proxy: proxy,
  hackney: [pool: false],
  ssl: [{:versions, [:'tlsv1.2']}]
])

edited Mar 19 '19 at 00:48

Paweł Obrok

22,568
8
74
70

answered Mar 18 '19 at 19:34

Nema Ga

2,450
4
26
49

That way you basically make your code single-threaded, which is not what most people expect from erlang. – Aleksei Matiushkin Mar 19 '19 at 06:45
@AlekseiMatiushkin can you please explain? I spawn a bunch of unlinked processes, each executing the code above and sending results back. Not sure if this is proper multi-threaded execution but it works. – Nema Ga Mar 19 '19 at 23:18
@AlekseiMatiushkin disabling the pool doesn't mean it's single-threaded, it'll still create separate connections, it just doesn't keep them around. – Johanna Larsson Mar 20 '19 at 20:05

HTTPoison manually close connection

2 Answers2