0

I am attempting to develop a small custom HTTP server application in Python 3, built around the http.server and socketserver libraries. Smaller files are served okay, but larger files keep failing. Calls to the wfile.write seem to be ignored after a random number of calls. I have written a very basic troubleshooting script which serves one preset file no matter the HTTP request.

port = 8080
host = "localhost"
filename = "test.txt"

import os
import http.server

class TestHTTPRequestHandler(http.server.BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"
    
    def do_GET(self):
        fileSize = os.path.getsize(filename)
        print("File size: %s" % fileSize)
        self.send_response(200)
        self.send_header("Content-Length", str(fileSize))
        self.send_header("Content-Type", "text/plain")
        self.end_headers()
        self.flush_headers()
        
        with open(filename, "rb") as f:
            totalBytesRead = 0
            totalBytesSent = 0
            while True:
                data = f.read(4096)
                if len(data) < 1: break
                totalBytesRead += len(data)
                totalBytesSent += self.wfile.write(data)
                self.wfile.flush()
            print("Total bytes read: %s" % totalBytesRead)
            print("Total bytes sent: %s" % totalBytesSent)
        

srv = http.server.HTTPServer((host, port), TestHTTPRequestHandler)
try:
    srv.serve_forever()
except KeyboardInterrupt:
    srv.server_close()

The output of this script on serving one request:

File size: 257754
127.0.0.1 - - [19/Mar/2022 18:41:24] "GET / HTTP/1.1" 200 -
Total bytes read: 257754
Total bytes sent: 257754

The number of bytes read and sent both match the size of the file, indicating that, at least within the program, the entire file was sent. However, curl disagrees:

$ curl -v http://localhost:8080/ > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:8080...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: BaseHTTP/0.6 Python/3.8.10
< Date: Sun, 20 Mar 2022 00:41:39 GMT
< Content-Length: 257754
< Content-Type: text/plain
< 
{ [12288 bytes data]
100  251k  100  251k    0     0  5993k      0 --:--:-- --:--:-- --:--:-- 5993k
* Connection #0 to host localhost left intact

Curl reports having received 12288 bytes (4096 * 3) out of the full 257754, and is inconsistent between requests. Requests from the browser usually either don't complete (if connection: keep-alive is set) or result in an incomplete file. I have tried using self.request.sendall() in place of self.wfile.write(), with the same results.

Oddly, Python3's http.server has the same issue when run using "python3 -m http.server". I have tried localhost, 127.0.0.1, and the interface's network address, and different ports, and all have the same issue.

Is this a problem with my code, or some sort of network configuration issue? I am running Python 3.8.10 on Ubuntu 20.04.

Josu
  • 46
  • 3
  • do you have this problem when you send all file at once - without `while True` loop? – furas Mar 20 '22 at 01:52
  • do you have this problem if you test with command `wget` or open in web browser? – furas Mar 20 '22 at 01:53
  • Possibly relevant https://stackoverflow.com/a/7647695/494134 – John Gordon Mar 20 '22 at 02:01
  • I get inconsistent, generally incorrect behavior in a browser. When reading and sending all at once, curl consistently reports receiving only 32768 bytes. Wget is working fine and saving the whole file with or without the loop. – Josu Mar 20 '22 at 03:31

0 Answers0