Accessing Mt Gox API via http-conduit-0.1.9.3: query string causes timeout

Question

I'm trying to access the Mt Gox REST API using http-conduit. Queries that just have a path (e.g. https://data.mtgox.com/api/2/BTCUSD/money/ticker) work fine, but when I add a queryString to the request it times out.

So this works:

mtGoxRequest :: String -> QueryText -> Request m
mtGoxRequest p qt = def {
   secure = True,
   host = "data.mtgox.com",
   port = 443,
   method = "GET",
   path = fromString $ "api/2/" ++ p,
   queryString = renderQuery False $ queryTextToQuery qt,
   responseTimeout = Just 10000000
}

currencyTicker :: Request m
currencyTicker = mtGoxRequest "BTCUSD/money/ticker" []

But this times out:

tradeStream :: Currency -> UTCTime -> Request m
tradeStream t = mtGoxRequest
   "BTCUSD/money/trades/fetch"
   [("since", Just $ T.pack $ utcToGoxTime t)]

The difference seems to be in the use of a queryString: when I added the bogus query "foo=bar" to the currencyTicker that timed out too.

However all this works fine in the web browser: going to https://data.mtgox.com/api/2/BTCUSD/money/ticker?foo=bar instantly returns the correct error message instead of timing out. The trade fetch URL works as well, although I won't include a link because the "since" argument says how far back to go. Conversely, if I remove the queryString from the trades list request it correctly returns the entire available trade history.

So something about the http-conduit query string is obviously different. Anyone know what it might be?

Here is the Haskell Request object being sent (as printed by "Show"):

Request {
  host                 = "data.mtgox.com"
  port                 = 443
  secure               = True
  clientCertificates   = []
  requestHeaders       = []
  path                 = "api/2/BTCUSD/money/trades/fetch"
  queryString          = "since=1367142721624293"
  requestBody          = RequestBodyLBS Empty
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = Just 10000000
}

According to its returned headers Mt Gox is using cloudflare-nginx and PHP 5.

Edit: Forgot to mention that when I use http-conduit to send a request with a queryString to http://scooterlabs.com/echo I get the correct response as well, so it seems to be some interaction between the Mt Gox webserver and http-conduit.

score 1 · Answer 1 · answered Apr 28 '13 at 20:46

1

Got it figured out. You need to add a User-Agent string. So

requestHeaders = [(CI.mk "User-Agent", "Test/0.0.1")],

in the middle of the request function makes it work.

answered Apr 28 '13 at 20:46

Paul Johnson

17,438
3
42
59

score 0 · Answer 2 · answered Apr 28 '13 at 15:26

0

$ time curl https://data.mtgox.com/api/2/BTCUSD/money/trades/fetch?since=1367142721624293
...
real    0m20.993s

Looks to me like everything is working correctly: the API call takes a while to return, so http-conduit throws a timeout exception since 20s is longer than 10s.

answered Apr 28 '13 at 15:26

Michael Snoyman

31,100
3
48
77

That takes only half a second for me. Meantime I've tried extending the timeout to 1 minute. No luck. Also this wouldn't explain the fact that requests without queryStrings work fine. – Paul Johnson Apr 28 '13 at 16:03
Did you try the actual URL I provided here? This isn't about arbitrary query strings, it seems to be that mtgox takes a while to process that specific request. – Michael Snoyman Apr 28 '13 at 19:25
Yes I did. Even though I tried it about 20 minutes after you, so the result set was bigger, it still only took half a second. I know that Mt Gox has been taking a lot of DDOS attacks, including at the application level, so I wonder if maybe I'm being blackholed by something at their end. – Paul Johnson Apr 28 '13 at 20:10

Accessing Mt Gox API via http-conduit-0.1.9.3: query string causes timeout

2 Answers2