137

Using Apache or Ngnix I always create development sites based on real projects such as http://project1.loc which, after adding to my .hosts file, the browser has no problem using.

However, when I attempt to make a cURL request (http://project1.loc/post.json) to that same URL I never get anything but a timeout. I'm assuming cURL doesn't care about my custom hosts and goes straight to a name server for it's info.

How can I fix this?

UPDATE I set a custom header "HOST: http://project1.loc" and now I am getting 400 errors - but they are instantaneous so I'm assuming that cURL is at least using the hosts file...

Xeoncross
  • 55,620
  • 80
  • 262
  • 364

7 Answers7

497

Actually, curl has an option explicitly for this: --resolve

Instead of curl -H 'Host: yada.com' http://127.0.0.1/something

use curl --resolve 'yada.com:80:127.0.0.1' http://yada.com/something

What's the difference, you ask?

Among others, this works with HTTPS. Assuming your local server has a certificate for yada.com, the first example above will fail because the yada.com certificate doesn't match the 127.0.0.1 hostname in the URL.

The second example works correctly with HTTPS.

In essence, passing a "Host" header via -H does hack your Host into the header set, but bypasses all of curl's host-specific intelligence. Using --resolve leverages all of the normal logic that applies, but simply pretends the DNS lookup returned the data in your command-line option. It works just like /etc/hosts should.

Note --resolve takes a port number, so for HTTPS you would use

curl --resolve 'yada.com:443:127.0.0.1' https://yada.com/something

John Hart
  • 5,718
  • 2
  • 20
  • 12
  • 30
    This is killing me - can someone please mark this as the right answer? It's much newer than the answer, so doesn't have the votes .. but the accepted answer is wrong (ie, only works for certain situations) =( – John Hart Aug 02 '12 at 16:50
  • This is indeed a great answer, and received my vote. Only Xenocross can mark an answer as accepted. In time others will likely come here and gradually vote yours higher. – hobodave Oct 19 '12 at 17:50
  • 11
    Worth noting that --resolve was only added in curl 7.21.3 - if you're stuck on an older host (eg Ubuntu 10.04 LTS) then the -H 'Host...' option is still a useful fallback. – Ken Nov 21 '12 at 10:20
  • Is there a way to make this true temporarily for all applications? Almost like a dynamic hosts file. – CMCDragonkai Jun 12 '14 at 07:11
  • 9
    While I agree this probably should be the accepted answer (and I would certainly not take offence if the OP changed it, on the contrary), saying that my answer is wrong isn't right: it is correct for the versions available at the time the question and answer was produced. SO users you can't be bothered reading past the first answer and also evaluate answers based on their timestamps will never get the best help... – Bruno Oct 26 '14 at 14:36
  • 1
    Sorry, Bruno, no offense meant. – John Hart Nov 10 '14 at 17:06
  • I had to remove quotes aroung 'yada.com:80:127.0.0.1' to have this command word otherwise it just call the url without using specified host,port and ip. Working command for me was : curl --resolve 'yada.com:80:127.0.0.1' http://yada.com/something both on linux and windows – Pansoul Feb 14 '19 at 13:44
121

EDIT: While this is currently accepted answer, readers might find this other answer by user John Hart more adapted to their needs. It uses an option which, according to user Ken, was introduced in version 7.21.3 (which was released in December 2010, i.e. after this initial answer).


In your edited question, you're using the URL as the host name, whereas it needs to be the host name only.

Try:

curl -H 'Host: project1.loc' http://127.0.0.1/something

where project1.loc is just the host name and 127.0.0.1 is the target IP address.

(If you're using curl from a library and not on the command line, make sure you don't put http:// in the Host header.)

Community
  • 1
  • 1
Bruno
  • 119,590
  • 31
  • 270
  • 376
  • 1
    I'm getting 400 errors with PHP and when I manually make the request with curl.exe I get the default index of the server which means it's not respecting the `HOST` header. – Xeoncross Aug 11 '10 at 15:29
  • I've tried it on various servers with virtual hosts, and it works (from the command line). Try `Host` not `HOST` just in case (although I think it shouldn't be case-sensitive). As I said, make sure you're using only the hostname in the `Host` header, nothing else (no `http://` and no `/something` after). How did you set up your hosts file? – Bruno Aug 11 '10 at 15:37
  • Posted more data about the results of doing this below. – Xeoncross Aug 11 '10 at 15:44
  • 1
    As Bruno said below, the problem is probably just my server config as the request seems to be making it and receiving a 403 error. – Xeoncross Aug 11 '10 at 16:21
  • I missed "127.0.0.1 myvirtualhost.localhost" in hosts file hence the problem. – Arvind K. Nov 20 '15 at 07:39
  • the working command for me was : curl -H Host:project1.loc http://127.0.0.1/something ie without quote and consequently no space between Host and Hostname – Pansoul Feb 14 '19 at 13:41
5

For setting up virtual hosts on Apache http-servers that are not yet connected via DNS, I like to use:

curl -s --connect-to ::host-name: http://project1.loc/post.json

Where host-name ist the IP address or the DNS name of the machine on which the web-server is running. This also works well for https-Sites.

effhaa
  • 51
  • 1
  • 3
  • 1
    This post was send 10 years ago and fixed via comments, thanks for contribution. In this kind of posts please check for answers, if there is an asnwear send up vote else post new answer because you could marked as spam. – Cafer Yükseloğlu May 07 '20 at 20:29
  • Interesting point. Old answers that have visibility (and are still pretty ok), may bury newer answers that aren't that bad either. A problem of vote/credit/reputation based websites. Thank you for your suggestion, --connect-to works indeed – Pier A Nov 07 '22 at 17:02
2

Either use a real fully qualified domain name (like dev.yourdomain.com) that pointing to 127.0.0.1 or try editing the proper hosts file (usually /etc/hosts in *nix environments).

Oli
  • 235,628
  • 64
  • 220
  • 299
  • I develop on windows using `system32/drivers/etc/hosts` – Xeoncross Aug 02 '10 at 18:26
  • Are you using the native build of cURL or some cygwin cross-build? I say this because I'm not sure how each resolve their DNS. The native *should* pick up from Windows' hosts file but a cygwin version might want a cygwin version. Either way, using a real domain pointing to 127.0.0.1 would work however things are set up. – Oli Aug 04 '10 at 21:19
  • I'm using the native windows build included with PHP 5.3 for windows (Running as php_fastcgi). – Xeoncross Aug 05 '10 at 15:42
2

It seems that this is not an uncommon problem.

Check this first.

If that doesn't help, you can install a local DNS server on Windows, such as this. Configure Windows to use localhost as the DNS server. This server can be configured to be authoritative for whatever fake domains you need, and to forward requests on to the real DNS servers for all other requests.

I personally think this is a bit over the top, and can't see why the hosts file wouldn't work. But it should solve the problem you're having. Make sure you set up your normal DNS servers as forwarders as well.

OmarOthman
  • 1,718
  • 2
  • 19
  • 36
hookenz
  • 36,432
  • 45
  • 177
  • 286
1

Does the server actually get the requests, and are you handling the host name (alias) properly?

after adding to my .hosts file

Check your webserver log, to see how the request came in...

curl has options to dump the request sent, and response received, it is called trace, which will will be saved to a file.

--trace

If you are missing host or header information - you can force those headers with the config option.

I would get the curl request working on the command line, and then try to implement in PHP.

the config option is

-K/--config

the options that are relevant in curl are here

--trace Enables a full trace dump of all incoming and outgoing data, including descriptive information, to the given output file. Use "-" as filename to have the output sent to stdout.

      This option overrides previous uses of -v/--verbose or --trace-ascii.

      If this option is used several times, the last one will be used.

-K/--config Specify which config file to read curl arguments from. The config file is a text file in which command line arguments can be written which then will be used as if they were written on the actual command line. Options and their parameters must be specified on the same config file line, separated by whitespace, colon, the equals sign or any combination thereof (however, the preferred separa- tor is the equals sign). If the parameter is to contain whitespace, the parameter must be enclosed within quotes. Within double quotes, the following escape sequences are available: \, \", \t, \n, \r and \v. A backslash preceding any other letter is ignored. If the first column of a config line is a '#' character, the rest of the line will be treated as a comment. Only write one option per physical line in the config file.

      Specify the filename to -K/--config as '-' to make curl read the file from stdin.

      Note that to be able to specify a URL in the config file, you need to specify it using the --url option, and not by simply writing the URL on its own line. So, it could look similar to this:

      url = "http://curl.haxx.se/docs/"

      Long option names can optionally be given in the config file without the initial double dashes.

      When curl is invoked, it always (unless -q is used) checks for a default config file and uses it if found. The default config file is checked for in the following places in this order:

      1) curl tries to find the "home dir": It first checks for the CURL_HOME and then the HOME environment variables. Failing that, it uses getpwuid() on UNIX-like systems (which  returns  the  home  dir
      given the current user in your system). On Windows, it then checks for the APPDATA variable, or as a last resort the '%USERPROFILE%\Application Data'.

      2)  On windows, if there is no _curlrc file in the home dir, it checks for one in the same dir the curl executable is placed. On UNIX-like systems, it will simply try to load .curlrc from the deter-
      mined home dir.

      # --- Example file ---
      # this is a comment
      url = "curl.haxx.se"
      output = "curlhere.html"
      user-agent = "superagent/1.0"

      # and fetch another URL too
      url = "curl.haxx.se/docs/manpage.html"
      -O
      referer = "http://nowhereatall.com/"
      # --- End of example file ---

      This option can be used multiple times to load multiple config files.
George Lambert
  • 616
  • 3
  • 4
  • Again, I'm using PHP on windows to fetch a page on a vhost on the same windows running nginx. Anyway, I made a request to a vhost `http://domain.loc/users/getSettings.xml` and this is what the access.log showed `127.0.0.1 - - [09/Aug/2010:11:42:55 -0500] "POST /users/getSettings.xml HTTP/1.1" 499 0 "-" "-"` and curl reported `Operation timed out after 10000 milliseconds with 0 bytes received` So I guess that cURL is actually handling the vhost since the access.log shows the request. Then again, it might now be making it to the correct domain... – Xeoncross Aug 09 '10 at 16:46
  • The "499 0" on that line is VERY significant. The process returned zero bytes - which curl was waiting for. and returned an HTTP 499 - which is a strange result. call another script - that returns a static string in response to the post - and see that you are getting the response in curl. You many not be posting the data as you expect... and the script may be timing out waiting for the response. also change the script to log the input to a temp file, and see that you are "receiving the expected post from your curl request" – George Lambert Aug 09 '10 at 18:14
  • Add did you try command line curl - so that you could control the post and see the server response? – George Lambert Aug 09 '10 at 18:15
  • Quick answer to second question - no. I don't know how to access the command line cURL on windows since it's built into PHP and not the windows terminal. – Xeoncross Aug 09 '10 at 19:50
  • 1
    you can download a command line version of curl for windows from here http://curl.haxx.se/download.html – George Lambert Aug 09 '10 at 21:58
  • Yep, straight curl works great on my PC. Thanks for the download link. curl.exe is *really* useful! No trouble at all with making the same requests PHP fails at. It must be something PHP is doing before to alter how cURL performs or something. – Xeoncross Aug 10 '10 at 15:17
  • So the question is - "what is the PHP actually sending" and why is it failing. Can you change your script to save the contents of the "actual" request to a file, and then look at the file contents... your post data may be getting malformed. – George Lambert Aug 10 '10 at 19:11
1

Making a request to

C:\wnmp\curl>curl.exe --trace-ascii -H 'project1.loc' -d "uuid=d99a49d846d5ae570
667a00825373a7b5ae8e8e2" http://project1.loc/Users/getSettings.xml

Resulted in the -H log file containing:

== Info: Could not resolve host: 'project1.loc'; Host not found
== Info: Closing connection #0
== Info: About to connect() to project1.loc port 80 (#0)
== Info:   Trying 127.0.0.1... == Info: connected
== Info: Connected to project1.loc (127.0.0.1) port 80 (#0)
=> Send header, 230 bytes (0xe6)
0000: POST /Users/getSettings.xml HTTP/1.1
0026: User-Agent: curl/7.19.5 (i586-pc-mingw32msvc) libcurl/7.19.5 Ope
0066: nSSL/1.0.0a zlib/1.2.3
007e: Host: project1.loc
0092: Accept: */*
009f: Content-Length: 45
00b3: Content-Type: application/x-www-form-urlencoded
00e4: 
=> Send data, 45 bytes (0x2d)
0000: uuid=d99a49d846d5ae570667a00825373a7b5ae8e8e2
<= Recv header, 24 bytes (0x18)
0000: HTTP/1.1 403 Forbidden
<= Recv header, 22 bytes (0x16)
0000: Server: nginx/0.7.66
<= Recv header, 37 bytes (0x25)
0000: Date: Wed, 11 Aug 2010 15:37:06 GMT
<= Recv header, 25 bytes (0x19)
0000: Content-Type: text/html
<= Recv header, 28 bytes (0x1c)
0000: Transfer-Encoding: chunked
<= Recv header, 24 bytes (0x18)
0000: Connection: keep-alive
<= Recv header, 25 bytes (0x19)
0000: X-Powered-By: PHP/5.3.2
<= Recv header, 56 bytes (0x38)
0000: Set-Cookie: SESSION=m9j6caghb223uubiddolec2005; path=/
<= Recv header, 57 bytes (0x39)
0000: P3P: CP="NOI ADM DEV PSAi COM NAV OUR OTRo STP IND DEM"
<= Recv header, 2 bytes (0x2)
0000: 
<= Recv data, 118 bytes (0x76)
0000: 6b
0004: <html><head><title>HTTP/1.1 403 Forbidden</title></head><body><h
0044: 1>HTTP/1.1 403 Forbidden</h1></body></html>
0071: 0
0074: 
== Info: Connection #0 to host project1.loc left intact
== Info: Closing connection #0

My hosts file looks like:

# Copyright (c) 1993-1999 Microsoft Corp.
#
# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to host names. Each
# entry should be kept on an individual line. The IP address should
# be placed in the first column followed by the corresponding host name.
# The IP address and the host name should be separated by at least one
# space.
#
# Additionally, comments (such as these) may be inserted on individual
# lines or following the machine name denoted by a '#' symbol.
#
# For example:
#
#      102.54.94.97     rhino.acme.com          # source server
#       38.25.63.10     x.acme.com              # x client host

127.0.0.1       localhost
...
...
127.0.0.1   project1.loc
Xeoncross
  • 55,620
  • 80
  • 262
  • 364
  • 1
    `-H` is for the full header, not just host, so use `-H 'Host: project1.loc'`. In addition, despite this problem, this request seems to work on the correct host (obtained correctly from your `hosts` file by curl on the command line at least). What's not working (403) seems like an authentication/authorization issue, so your server seems to be blocking these requests. I'd suggest fixing the server configuration for this. – Bruno Aug 11 '10 at 15:52