Proxy server made with sockets (python) receiving and sending in incorrect order

Question

I was trying to make a simple proxy server, that can be used to relay any request message to some destination host, and receive the response back and relay it back to the original sender (client). My implementation causes the response of the first request message to get stalled in the server, and it only gets received as the response of the second request message. This causes some errors. I was trying to use this as an http proxy, running a simple http server on localhost, the proxy also on localhost, and used my browser to send GET messages to the proxy.

Below is the code

class Proxy:
    def __init__(self,mhost,mport,dhost,dport):
        self.mhost = mhost
        self.mport = mport
        self.dhost = dhost
        self.dport = dport

        #self.middle_man = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
        #self.middle_man.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

        self.colorAndFormat = ColorAndFormat() # Just for coloring the output text

    def start_listener(self):
        colorAndFormat = ColorAndFormat()

        middle_man_recv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        middle_man_recv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        middle_man_recv.bind((self.mhost, self.mport))
        middle_man_recv.listen()
        print(colorAndFormat.Blue2("Listening on " + self.mhost + ':' + str(self.mport) + ' ...\n---------------------------------'))

        middle_man_forw = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        middle_man_forw.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        #middle_man_forw.connect_ex((self.dhost, self.dport))

        while True:
            #try:
            # Receive data
            client_socket,client_addr = middle_man_recv.accept()
            data = client_socket.recv(1024)
            print(colorAndFormat.Blue2("Data received from client " + client_addr[0] + ':' + str(client_addr[1])))
            print(colorAndFormat.Blue2(data.decode()))

            # Forward data to dhost:dport
            middle_man_forw.connect_ex((self.dhost, self.dport))
            middle_man_forw.sendall(data)
            print(colorAndFormat.Beige("Data forwarded to " + self.dhost + ':' + str(self.dport)))

            # Receive data from dhost:dport
            data = middle_man_forw.recv(1024)
            print(colorAndFormat.Beige("Data received from " + self.dhost + ':' + str(self.dport)))
            print(colorAndFormat.Beige(data.decode()))

            # Forward data to client
            client_socket.sendall(data)
            print(colorAndFormat.Blue2("Data forwarded to client"))
            print('---------------------------------')
            client_socket.close()

            #except

Below is the output on sending a GET request. (index.html exists)

Listening on 127.0.0.1:4444 ...
---------------------------------
Data received from client 127.0.0.1:60032
GET /index.html HTTP/1.1
Host: 127.0.0.1:4444
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Upgrade-Insecure-Requests: 1


Data forwarded to 127.0.0.1:8888
Data received from 127.0.0.1:8888
HTTP/1.0 200 OK
Server: SimpleHTTP/0.6 Python/3.8.3
Date: Tue, 17 Nov 2020 20:13:53 GMT
Content-type: text/html
Content-Length: 116
Last-Modified: Tue, 17 Nov 2020 19:49:32 GMT


Data forwarded to client
---------------------------------
Data received from client 127.0.0.1:60036
GET /favicon.ico HTTP/1.1
Host: 127.0.0.1:4444
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0
Accept: image/webp,*/*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive


Data forwarded to 127.0.0.1:8888
Data received from 127.0.0.1:8888
<html>
<head><title>This worksss!!</title></head>
<body>
If you're seeing this, the Proxy works!!                                      
</body>                                                                         
</html>                                                                         
                                                                                
Data forwarded to client
---------------------------------
Data received from client 127.0.0.1:60038
GET /index.html HTTP/1.1
Host: 127.0.0.1:4444                                                            
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8                                                                              
Accept-Language: en-US,en;q=0.5                                                 
Accept-Encoding: gzip, deflate                                                  
Connection: keep-alive                                                          
Upgrade-Insecure-Requests: 1                                                    
                                                                                
                                                                                
Traceback (most recent call last):
  File "exposer.py", line 176, in <module>
    exposer.start_listener()
  File "exposer.py", line 40, in start_listener
    middle_man_forw.sendall(data)
BrokenPipeError: [Errno 32] Broken pipe

Is this implementation wrong?

https://stackoverflow.com/a/43420503/238704 – President James K. Polk Nov 18 '20 at 01:04 — President James K. Polk, Nov 18 '20 at 01:04

score 0 · Answer 1 · answered Nov 17 '20 at 21:16

        data = middle_man_forw.recv(1024)

You expect data to contain the full response of the server. This expectation is wrong. First, it might be more than the 1024 bytes you are trying to read. Then, recv might only return a part of the data, even if the total data are less than 1024 bytes.

The reason for this is that TCP is not a message protocol but just a byte stream. But you expect to recv full messages, with the assumption that a HTTP response is a single message.

There are basically two approaches on how to handle this: you can continue with your approach where you first expect the HTTP request, forward it, expect the HTTP response and forward it. But then you need to make sure that you actually read full requests and responses, i.e. you have to understand and properly implement the HTTP protocol.

The other approach would be to simply forward everything from client to server and in parallel forward everything from server to client. In this case you don't actually care what protocol is spoken but just forward everything blindly. This approach can be implemented with threads (one thread for each direction), select or similar. See for example A python proxy in less than 100 lines of code for the select-based approach.

Proxy server made with sockets (python) receiving and sending in incorrect order

1 Answers1