Can't fetch desired content using socket

Question

I'm trying to get the visible content from here using socket but unfortunately I'm getting an error when I execute my script. As I'm very new to code using socket, I can't understand as to where I'm going wrong.

My code:

import socket

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host_ip = socket.gethostbyname('data.pr4e.org')
s.connect((host_ip,80))
cmd = "GET http://data.pr4e.org/romeo.txt HTTP/1.0\n\n".encode()
s.send(cmd)

while True:
    data = s.recv(1024)
    if (len(data) <1 ):
        break
    print(data.decode())
s.close()

Error I'm getting:

400 Bad Request

Your browser sent a request that this server could not understand.

This error comes from the server. Your script is correctly reading the response. — zvone, Oct 27 '18 at 20:49
[Here](https://stackoverflow.com/q/6686261/238704) is a relevant SO question. — President James K. Polk, Oct 28 '18 at 00:39

score 1 · Accepted Answer · answered Oct 27 '18 at 20:55

1

I was able to obtain the desired result by adding \r\n\r\n to the end of the request command, rather than the original \n\n:

import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((socket.gethostbyname('data.pr4e.org'), 80))
s.sendall("GET http://data.pr4e.org/romeo.txt HTTP/1.0\r\n\r\n".encode())
print(s.recv(1024))

Output:

...
Content-Type: text/plain\r\n\r\nBut soft what light through yonder window breaks\nIt is the east and Juliet is the sun\nArise fair sun and kill the envious moon\nWho is already sick and pale with grief\n'

answered Oct 27 '18 at 20:55

Ajax1234

69,937
8
61
102

Just use the `requests` library, then you won’t also hit a host of other problems that `requests` has already solved. – DisappointedByUnaccountableMod Oct 28 '18 at 17:23
@barny I do agree, however, the OP is specifically trying to use `socket`. – Ajax1234 Oct 28 '18 at 17:24

score 1 · Answer 2 · edited Oct 07 '21 at 12:12

There are multiple problems here:

It is uncommon to include http://data.pr4e.org after GET (see RFC 7230) unless talking to a proxy. You will usually write GET /romeo.txt and provide the hostname in a separate Host: data.pr4e.org header. Servers are required to support the form you used, but they might violate the standard and choke on it. This is especially likely if you claim to be using HTTP/1.0, which is stricter and forbids this form unless talking to a proxy.
Nobody uses HTTP/1.0 any more. All modern browsers and other HTTP clients use HTTP/1.1 or HTTP/2. Some servers will support HTTP/1.0, but it's not mandatory. Note that HTTP/1.1 makes the Host: header mandatory, even when you put the full URL after GET.
HTTP/1.0 uses \r\n ("CRLF") as a newline (see RFC 1945), so \n may not always be understood. Again, some servers will handle it correctly, but it is non-conforming. The use of CRLF has been carried over to HTTP/1.1.
print(data.decode()) will add an extra newline at the end of data. This could become an issue if TCP fragments a large HTTP response so that recv() returns multiple nonempty strings. Use print(data.decode(), end='') instead.

This is all perfectly valid critique of the question, but that doesn’t make it an answer. — DisappointedByUnaccountableMod, Oct 28 '18 at 17:22
@barny: The `\r\n` issue is the cause of OP's particular error in this case, so I don't see how this is "not an answer" at all. — Kevin, Oct 28 '18 at 17:23

Can't fetch desired content using socket

2 Answers2