Why do I get a timeout in PhpStorm HTTP Client and Laravel HTTP Client but not in Postman?

Question

When I send a plain GET request with Postman this takes about 228ms consistently and yields a json response. As you can see I disabled cookies and headers in the request to have the exact same request. The generated curl seems to confirm this. Please have a better look at my first 2 screenshots of Postman for proof (I would doubt me as well). No headers at all, even in de 2nd image of the debug console you'll see no headers should be sent.

When I try the same thing with PhpStorm's HTTP client

I get a timeout (except for 1 time which worked after 30 seconds)

Same thing in code:

$result = Http::timeout(10)
    ->get('https://mobileapi.jumbo.com/v17/products');


dump($result->body());

Making the request in curl is even weirder:

So we have 3 different responses for seemingly the same request:

Postman: fast response (< 400ms) with expected body
both PhpStorm and php code: timeout even with more that 10 seconds of allowed time
curl request generated by Postman: Access Denied

Update Found a solution to my problem, but the reason why Postman works without it still baffles me: A User-Agent header containing exactly "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Firefox/102.0" works. However as you can see in above screenshots I don't send this header in Postman and there it works without. This question as to why this happens still stands.

could be just request headers, maybe `Accept`, `Content-Type` , or SSL verification, you can add `clockwork` on that application and inspect the header of each request for those that work vs those that doesn't work — silver, Jul 02 '23 at 04:01
In visual studio code exists an extension called `Thunder Client` (to make request, similar to postman), the results that I got from this extension took between 2s and 9s. The curl problem, it may be caused because you didn't define a user agent, I will add a link to a post of SO, you will find there a better explanation: https://stackoverflow.com/a/59809974/18618878, but in short you need to try and run the curl command specifing a user agent: `curl --user-agent 'Chrome/79' https://mobileapi.jumbo.com/v17/products`. — haruk1515, Jul 02 '23 at 06:52
I made use of wget on `powershell` too, and it worked, the response to retrive the data took me around 4s - 9s. @silver I don't think the problem is coming from the headers, neither from the cookies. For the headers, I tested with: `Content-Type,User-Agent,Content-Length,x-request-id,x-jumbo-store,jmb-service,Connection,Strict-Transport-Security,accept-encoding`. And for the Cookies I added the ones specified on postman with they respective domain: `user-session -> mobileapi.jumbo.com`, `_abck -> jumbo.com` -> `bm_sz -> jumbo.com` However i still got a 403 Forbidden on the php response. — haruk1515, Jul 02 '23 at 06:52
@haruk1515 I don't send any additional headers (so also no user agent), so I'm afraid that's not the solution — online Thomas, Jul 02 '23 at 12:19
@onlineThomas there are different headers auto added based on how you send the request, if you compare Postman, Curl, VSCode REST Client, Laravel HTTP(Guzzle) or Simply opening a URL in the browser. each of them have different headers, the most obvious one would be the `user-agent` which may or may not affect the response. in your case, its highly likely that there's missing required headers on those request that did not work, and your request could have been terminated by the web server before it actually reaches the actual application. that or even the inital handshake fails. — silver, Jul 02 '23 at 12:51
you can simply open the URL in the browser (if it works), check and copy all the request headers (you'll see them on the network tab), replicate it on your CURL request and remove one by one until it no longer works. — silver, Jul 02 '23 at 12:51
To slove part of the mystery: `curl --location 'https://mobileapi.jumbo.com/v17/products' --header "User-Agent: "` works in CURL meaning the API is explicitly blocking any `curl` user agents. The time it takes me varies but is consistnetly well above the <400ms mentioned in your post. I've seen times between 5 and up to 25 seconds to get a response, so the API may be caching responses based on something that Postman sends (maybe the postman token) that other clients don't send. I think the last part only the API providers can answer — apokryfos, Jul 03 '23 at 08:15
Are you using a CDN like Cloudflare? I noticed that when I have a domain going trough Cloudflare: Postman or curl requests are denied. So the CDN is applying some additional security checks. — Mtxz, Jul 04 '23 at 12:59
I face this problem before in windows but I fix it by moving to Linux and works fine for me — Joseph, Jul 04 '23 at 16:06
Do you use any VM or containers? It may not be obvious, but for example also WSL on Windows is "VM" = it has different networking. Do you use any proxy? Again, it may not be obvious, but especially corporate environment has access to internet only via proxy. That can be preconfigured via env variables or IT department can manage it on your machine. — Jan Garaj, Jul 04 '23 at 18:55
can you access your local web server log to see if you find anything? PHP log also. — Mtxz, Jul 05 '23 at 00:53
Could you do low level network sniffing (Wireshark) to see, what is source/destination for those Postman, PhpStorm, curl requests. Is firewall enabled? — Jan Garaj, Jul 05 '23 at 07:35
Sniffing over TLS is quite complicated, an easier approach is to setup a web server or use an online service to capture requests — Mihail Feraru, Jul 08 '23 at 13:13
Please review *[Why not upload images of code/errors when asking a question?](https://meta.stackoverflow.com/questions/285551/)* (e.g., *"Images should only be used to illustrate problems that* ***can't be made clear in any other way,*** *such as to provide screenshots of a user interface."*) and [do the right thing](https://stackoverflow.com/posts/76581792/edit). Thanks in advance. — Peter Mortensen, Jul 22 '23 at 03:31

score 1 · Accepted Answer · answered Jul 08 '23 at 13:10

Why do you get different behaviors using seemingly the same request? Surely, the web server has some request filtering rules based on HTTP headers.

Even if it looks that all HTTP clients send the same request, the reality is that each one attaches slightly different headers silently. Given so, I created a RequestBin to check the differences between the clients.

curl --location 'https://...'

Host: envyr05unq3bk.x.pipedream.net
X-Amzn-Trace-Id: Root=1-64a95ba2-4633081a75af397c63ece198
User-Agent: curl/7.71.1
Accept: */*

Postman with Host header only

Host: envyr05unq3bk.x.pipedream.net
X-Amzn-Trace-Id: Root=1-64a95bac-102cc1e826a026f328c6c583

PhpStorm HTTP Client

Host: envyr05unq3bk.x.pipedream.net
X-Amzn-Trace-Id: Root=1-64a95cc5-40d9098a346629e74f52f4d9
User-Agent: Apache-HttpClient/4.5.14 (Java/17.0.7)
Accept-Encoding: br,deflate,gzip,x-gzip

Checking against your domain, it seems that setting the User-Agent to curl/7.71.1 or Apache-HttpClient/4.5.14 (Java/17.0.7) causes a time-out. As mentioned in other responses, there is probably a blacklist for user agents.

**Just in case anyone stumbles upon this**. The Trace ID has **nothing** to do with it, neither has the version of the User-Agent. The only piece of advice that is correct, is the last line, which mentions that this had already been answered in the other post that has been downvoted. — srn, Jul 08 '23 at 16:09

score 0 · Answer 2 · answered Jul 03 '23 at 08:02

0

It's often due to some subtle differences in the way the requests are being sent or received. Here are some possible reasons:

Headers: Postman might automatically include headers that PhpStorm, Laravel, or curl don't.

SSL/TLS configurations: If the server uses HTTPS, the SSL/TLS setup could cause differences.

Proxies: Differences in proxy settings across the clients could affect the results.

Cookies: Ensure cookies are truly disabled in all clients.

It's important to compare the actual requests being sent in each client to identify any discrepancies. One other possible reason can be Firewall Settings

answered Jul 03 '23 at 08:02

Muhammad Tahir Qaiser

570
7
26

Probably this. Postman does not "fake" HTTP-Requests to look like a Browser/Laraval/PHP-HTTP-Requests. – Hellma Jul 04 '23 at 09:37
Although it turns out that adding a header does solve the problem to get a proper response, this is kind of a general answer. The question is more about difference between the clients which appear to send the same request but probably ultimately don't. – online Thomas Jul 04 '23 at 13:59

score 0 · Answer 3 · answered Jul 04 '23 at 16:01

Postman has a default User-Agent that it sends with every request (temporary headers) if you don't specify a different one. This is likely why you're able to make the request in Postman without explicitly setting the User-Agent header. The api endpoint probably recognizes Postman's default User-Agent and allows the request.

You could try to enable the header in the header section but leave the value field blank. This will override the temp headers set by postman.

PhpStorm's built-in HTTP client does not automatically add a User-Agent header to the requests. If you want to include a User-Agent header, you need to add it manually to your request.

The api endpoint probably requires some headers to be set, otherwise it will block the request.

srn · Answer 4 · 2023-07-08T06:53:04.723

Weird indeed, but the answer is quite simple: The site uses a blacklist to filter User-Agent's and quite possible does so with a homemade solution.

First indicator is looking at their homepage's robots.txt, which may indicate that they've seen increased traffic due to ChatGPT bots or simply don't want to be scraped. It's quite "a bit" of an old-fashioned approach and the disallow-list in there looks like manually put together.

Second at least for the API they seem to parse the User-Agent and have some checks in place. Setting the UA to \0 for instance produces a 400 Bad Request reply.

Third, using Curl and setting the User-Agent to Cool returns a proper result:

$ curl --header "User-Agent: Cool" https://mobileapi.jumbo.com/v17/products

Fourth, the letters php seem to be on that blacklist, which means any User-Agent that starts with it (PhpStorm , phpstorm , php2020) seems to lead to no reply. User Agents ThePhp or ThisIsA Php Bot work though ...

However and finally, they also seem to use a whitelist. Setting headers to PhpStorm and the request times out. Setting it to Mozilla PhpStorm and a proper reply comes back. Not for PhpMozilla or Php Mozilla From that I can only assume they first check for known browser identifiers and reply if they find, then check for blacklisted strings (but without properly going through the substrings) and if they find neither, they still reply (which seems to be the case for Postman and my absurd examples).

For instance setting UA to StackOverflow also yields a good reply. Ah and guzzle seems to be on that blacklist as well (I guess that's your Laravel UA).

TL;DR Postman works, with the headers you've set, because EVERY User Agent works that is not on the (weirdly parsing) blacklist.

On a serious note, someone should tell them about about rate limiting or some other ways to protect against over the top traffic.

Why do I get a timeout in PhpStorm HTTP Client and Laravel HTTP Client but not in Postman?

4 Answers4