Curl Getting HTTP Response 0 instead of 200

Question

Hi I have the following simple script to read some URLs from the text and post it to another text file with response.

#!/bin/bash

while read url

do

    urlstatus=$(curl -o /dev/null --silent --head --write-out '%{http_code}' "$url")
    
        echo "$url"
        echo "$url  $urlstatus" >> urlstatus.txt
        
        done < $1

As an example I am trying following link:

www.proddigia.com/inmueble/pisos/en-venta/el-putget-i-el-farro/sarria-sant-gervasi/barcelona/6761

However I get 0 as response. When I check with google I get 200. Am I missing something in script?

AND If I use the exact URL supplied above (with `< ... >` wrapper) I get `000`. Sorry I can't help more. Good luck. — shellter, Jul 11 '20 at 13:55
From curl man-page: -w, --write-out NOTE: The %-symbol is a special symbol in the win32-environment, where all occurrences of % must be doubled when using this option. — Léa Gris, Jul 11 '20 at 14:03
Also added chrome response, weird thing is that with google I recieve 200 without any problem buy in curl the same link does not work.. — nic, Jul 11 '20 at 14:29
Just a couple of things. Firstly, I have run the example you provided on a Linux machine, and got the same response as @oguz-ismail has mentioned before, an HTTP STATUS 301 response (a redirect). If you want to follow the redirect, and consequently get the 200 response code, you should use curl with the '-L' option. Secondly, to better replicate your problem, what is your execution environment? — Néstor Lucas Martínez, Jul 11 '20 at 14:39
`www.proddigia.com/inmueble/pisos/en-venta/el-putget-i-el-farro/sarria-sant-gervasi/barcelona/6761` is not an URL, or only part of it. It has no protocol indicator. When you paste use this in a Chrome browser, it dumbs down hides the true URL, the server gets first a default HTTP request, then `301` redirect to the same address with HTTPS. And you only get the https logged in Chrome.. As I already commented, if you are in a Windows environment, then double the `%` as `'%%{http_code}'` and it should do for you. — Léa Gris, Jul 11 '20 at 15:13

Stephen C · Answer 1 · 2020-07-11T14:02:46.140

1

Zero is not a valid HTTP response code.

If curl is unable to establish a HTTP connection to the server, or if the server (somehow) fails to deliver a well-formed HTTP response message, there will be no "http code" to return in that variable. Zero is what you would probably see in that scenario.

It could also be that the value of $url that you are using is invalid. For example, if the URL is enclosed in < and > characters, it curl won't understand it. I would expect a zero in that case too.

The problem is that --silent is telling curl to throw away all of the error messages, so it can't tell you what the problem is.

I suggest that you see what you get by running the following command:

  curl -o /dev/null --head "$url"

with the identical url string to the one you are currently using.

edited Jul 11 '20 at 14:02

answered Jul 11 '20 at 13:48

Stephen C

698,415
94
811
1,216

But in that case I would expect same response from chrome network tab right? I believe that somehow it gets timeout when it asks for a response. Because if I run without silent mode I get everything 0. – nic Jul 11 '20 at 14:35
I wouldn't necessarily expect that you would get the same result using `chrome` as with `curl`. The network proxy settings could be different. Agent strings could be different. You may not even be sending the HTTP request from the same machine. – Stephen C Jul 11 '20 at 14:41

score 0 · Answer 2 · answered Jul 11 '20 at 15:37

0

I just figured out that if you use txt file created in windows OS it does not work as expcted in ubuntu. That was the reason that I got 0. You need to create the txt file in Ubuntu and copy the links over there. Thanks for the answers anyway.

answered Jul 11 '20 at 15:37

nic

125
1
3
7

1

Sounds like a [line endings problem](https://stackoverflow.com/questions/39527571/are-shell-scripts-sensitive-to-encoding-and-line-endings). DOS/Windows uses an extra carriage return at the end of each line, unix tools treat that as part of the line's content, and chaos results. – Gordon Davisson Jul 11 '20 at 17:38
The `dos2unix` and `unix2dos` commands convert between Unix/Linux and Windows line separators. – Stephen C Jul 12 '20 at 02:26

Curl Getting HTTP Response 0 instead of 200

2 Answers2