Gun http client not printing response body

Question

I have a http gen_server that uses gun as the http client library. I am able to open a connection and make a GET request. However, I get no response body, but the message received is 'fin. How to get a gun response in asynchronous mode?

(defun get (url-map)
  (io:format "http get ~p~n" `(,url-map))
  (let* ((`#(ok ,con-pid) (gun:open (binary:bin_to_list (map-get url-map 'host))
                                    (if (== (map-get url-map 'scheme) #"https")
                                      443
                                      80)))
         (mon-ref (monitor 'process con-pid)))
    (receive
      (msg
       (progn
         (io:format "gun:open msg ~p~n" `(,msg))
         (handle-get-request url-map con-pid mon-ref)))
     ((after 3000) (exit 'timeout)))))

(defun handle-get-request (url-map con-pid mon-ref)
  (io:format "handle get request path: ~p~n" `(,(map-get url-map 'path))) 
  (let ((stream-ref (gun:get con-pid (map-get url-map 'path))))
    (receive
      ((tuple 'gun_response con-pid stream-ref 'fin status headers)
       (progn
         (io:format "get response body fin~nstatus ~p~nheaders ~p~n"
                    `(,status ,headers))
         'no_data))  ; <--- how to get the response body content?
      ((tuple 'gun_response con-pid stream-ref 'nofin status headers)
       (progn
         (io:format "get response body nofin status ~p~n" `(,status))
         (handle-get-response url-map con-pid mon-ref stream-ref)))
      ((tuple 'DOWN mon-ref 'process con-pid reason)
       (io:format "handle get request error ~p~n" `(,reason))
       (exit reason))
      ((after 5000) (exit 'timeout)))))


(defun handle-get-response (url-map con-pid mon-ref stream-ref)
  (io:format "handle-get-response~n")
  (receive
    ((tuple 'gun_data con-pid stream-ref 'nofin data)
     (progn
       (io:format "get response data ~p~n" `(,data))
       (handle-get-response url-map con-pid mon-ref stream-ref)))
    ((tuple 'gun_data con-pid stream-ref 'fin data)
     (io:format "handle get response finished ~p~n" `(,data)))
    ((tuple 'DOWN mon-ref 'process con-pid reason)
     (io:format "handle get request error ~p~n" `(,reason))
     (exit reason))
    ((after 5000) (exit 'timeout))))

The request log is

http cast get state: #{host => <<"httpstat.us">>,path => <<"/200">>,
                       scheme => <<"https">>}
http get #{host => <<"httpstat.us">>,path => <<"/200">>,scheme => <<"https">>}
gun:open msg {gun_up,<0.146.0>,http}
handle get request path: <<"/200">>
get response body fin  <--- how to get the response body content?
status 200
headers [{<<"cache-control">>,<<"private">>},
         {<<"server">>,<<"Microsoft-IIS/10.0">>},
         {<<"x-aspnetmvc-version">>,<<"5.1">>},
         {<<"access-control-allow-origin">>,<<"*">>},
         {<<"x-aspnet-version">>,<<"4.0.30319">>},
         {<<"x-powered-by">>,<<"ASP.NET">>},
         {<<"set-cookie">>,
          <<"ARRAffinity=93fdbab9d364704de8ef77182b4d13811344b7dd1ec45d3a9682bbd6fa154ead;Path=/;HttpOnly;Domain=httpstat.us:443">>},
         {<<"date">>,<<"Wed, 16 Oct 2019 19:07:29 GMT">>},
         {<<"content-length">>,<<"0">>}]

I am able to get response body for other urls. For example:

handle-post <<"http://date.jsontest.com/">>
dreamnet-svc-sup start child [#{url => <<"http://date.jsontest.com/">>}]
http cast get state: #{host => <<"date.jsontest.com">>,path => <<"/">>,
                       scheme => <<"http">>}
http get #{host => <<"date.jsontest.com">>,path => <<"/">>,
           scheme => <<"http">>}
gun:open msg {gun_up,<0.188.0>,http}
handle get request path: <<"/">>
get response body nofin status 200
handle-get-response
handle get response finished <<"{\n   \"date\": \"10-17-2019\",\n   \"milliseconds_since_epoch\": 1571333596738,\n   \"time\": \"05:33:16 PM\"\n}\n">>
lfe>

Looking further into this, using erlang's httpc client too, the body is empty.

lfe> (lists:map (lambda (a) (application:start a)) '(crypto asn1 public_key ssl inets))
lfe> (httpc:request 'get `#("https://httpstat.us/200" ()) '() '())
#(ok
  #(#("HTTP/1.1" 200 "OK")
    (#("cache-control" "private")
     #("date" "Thu, 17 Oct 2019 17:29:50 GMT")
     #("server" "Microsoft-IIS/10.0")
     #("content-length" "0")
     #("x-aspnetmvc-version" "5.1")
     #("access-control-allow-origin" "*")
     #("x-aspnet-version" "4.0.30319")
     #("x-powered-by" "ASP.NET")
     #("set-cookie"
       "ARRAffinity=93fdbab9d364704de8ef77182b4d13811344b7dd1ec45d3a9682bbd6fa154ead;Path=/;HttpOnly;Domain=httpstat.us"))
    ()))

Maybe it's some issue with the server, but I am able to get response using curl.

curl -X GET "https://httpstat.us/200"
200 OK%

7stud · Answer 1 · 2019-10-17T20:05:04.410

If you wanted to get the response synchronously, then after you call gun:get you can use the returned stream-ref to call gun:await. See the gun docs for gun:await. So, something like:

(let (stream-ref (gun:get con-pid (map-get url-map 'path))))
(receive
((tuple 'data 'fin data)
 (progn
   (io:format "get response data ~p~n" `(,data))

But, you don't want to wait for a response, so you might try looping over your original receive until you get the type of message that you want. You can put your original receive in a recursive function, then end the recursive function when you get the type of message that you want. In other words, you will still receive the 'fin message, but your function won't end there--it will keep receiving messages until it receives a 'data message.

But that makes the call blocking and I am looking for an async solution. — John Doe, Oct 17 '19 at 17:44

Gun http client not printing response body

1 Answers1