2

I have been trying to make requests to a website using the requests library but using different network interfaces. Following are a list of answers that I have tried to use but did not work.

This answer describes how to achieve what I want, but it uses pycurl. I could use pycurl but I have learned about this monkey patching thing and want to give it a try.

This other answer seemed to work at first, since it does not raise any error. However, I monitored my network traffic using Wireshark and the packets were sent from my default interface. I tried to print messages inside the function set_src_addr defined by the author of the answer but the message did not show up. Therefore, I think it is patching a function that is never called. I get a HTTP 200 response, which should not occur since I have bound my socket to 127.0.0.1.

import socket

real_create_conn = socket.create_connection

def set_src_addr(*args):
    address, timeout = args[0], args[1]
    source_address = ('127.0.0.1', 0)
    return real_create_conn(address, timeout, source_address)

socket.create_connection = set_src_addr

import requests
r = requests.get('http://www.google.com')
r

<Response [200]>

I have also tried this answer. I can get two kind of errors using this method:

import socket                  
true_socket = socket.socket    
def bound_socket(*a, **k):     
    sock = true_socket(*a, **k)
    sock.bind(('127.0.0.1', 0))
    return sock                
socket.socket = bound_socket   
import requests

This will not allow me to create a socket and raise this error. I have also tried to make a variation of this answer which looks like this:

import requests                           
import socket                             
true_socket = socket.socket               
def bound_socket(*a, **k):                
    sock = true_socket(*a, **k)           
    sock.bind(('192.168.0.10', 0))        
    print(sock)                           
    return sock                           
socket.socket = bound_socket              
r = requests.get('https://www.google.com')

This also do not work and raises this error.

I have the following problem: I want to have each process sending requests through a specific network interface. I thought that since threads share global memory (including libraries), I should change my code to work with processes. Now, I want to apply a monkey patching solution somewhere, in a way that each process can use a different interface for communication. Am I missing something? Is this the best way to approach this problem?

Edit: I also would like to know if it is possible for different process to have different versions of the same library. If they are shared, how can I have different versions of a library in Python (one for each process)?

alexandredias3d
  • 361
  • 3
  • 17

3 Answers3

1

This appears to work for python3:

In [1]: import urllib3

In [2]: real_create_conn = urllib3.util.connection.create_connection

In [3]: def set_src_addr(address, timeout, *args, **kw):
   ...:     source_address = ('127.0.0.1', 0)
   ...:     return real_create_conn(address, timeout=timeout, source_address=source_address)
   ...:
   ...: urllib3.util.connection.create_connection = set_src_addr
   ...:
   ...: import requests
   ...: r = requests.get('http://httpbin.org')

It fails with the following exception:

ConnectionError: HTTPConnectionPool(host='httpbin.org', port=80): Max retries exceeded with url: / (Caused by NewConnectionError("<urllib3.connection.HTTPConnection object at 0x10c4b89b0>: Failed to establish a new connection: [Errno 49] Can't assign requested address",))
salparadise
  • 5,699
  • 1
  • 26
  • 32
0

I will document the solution I have found and list some problems I had in the process.

salparadise had it right. It is very similar to the first answer I have found. I am assuming that the requests module import the urllib3 and the latter has its own version of the socket library. Therefore, it is very likely that the requests module will never directly call the socket library, but will have its functionality provided by the urllib3 module.

I have not noticed it first, but the third snippet I had in my question was working. The problem why I had a ConnectionError is because I was trying to use a macvlan virtual interface over a wireless physical interface (which, if I understood correctly, drops packets if the MAC addresses do not match). Therefore, the following solution does work:

import requests                                
from socket import socket as backup            
import socket                                  
def socket_custom_src_ip(src_ip):              
    original_socket = backup                   
    def bound_socket(*args, **kwargs):         
        sock = original_socket(*args, **kwargs)
        sock.bind((src_ip, 0))                 
        print(sock)                            
        return sock                            
    return bound_socket

In my problem, I will need to change the IP address of a socket several times. One of the problems I had was that if no backup of the socket function is made, changing it several times would cause an error RecursionError: maximum recursion depth exceeded. This occurs since on the second change, the socket.socket function would not be the original one. Therefore, my solution above creates a copy of the original socket function to use as a backup for further bindings of different IPs.

Lastly, following is a proof of concept of how to achieve multiple processes using different libraries. With this idea, I can import and monkey-patch each socket inside my processes, having different monkey-patched versions of them.

import importlib                                     
import multiprocessing                               
class MyProcess(multiprocessing.Process):            
    def __init__(self, module):                      
        super().__init__()                           
        self.module = module                         
    def run(self):                                   
        i = importlib.import_module(f'{self.module}')
        print(f'{i}')
p1 = MyProcess('os')                                                          
p2 = MyProcess('sys')                
p1.start()                                                                    
<module 'os' from '/usr/lib/python3.7/os.py'>
p2.start()                           
<module 'sys' (built-in)>

This also works using the import statement and global keyword to provide transparent access inside all functions as the following

import multiprocessing                   
def fun(self):                           
    import os                            
    global os                            
    os.var = f'{repr(self)}'             
    fun2()                               
def fun2():                              
    print(os.system(f'echo "{os.var}"')) 
class MyProcess(multiprocessing.Process):
    def __init__(self):                  
        super().__init__()               
    def run(self):                       
        if 'os' in dir():                
            print('os already imported') 
        fun(self)                                            
p1 = MyProcess()                                                              
p2 = MyProcess()                                                              
p2.start()                                                                  
<MyProcess(MyProcess-2, started)>                                 
p1.start()                                        
<MyProcess(MyProcess-1, started)>
alexandredias3d
  • 361
  • 3
  • 17
0

I faced a similar issue where I wanted to have some localhost traffic originating not from 127.0.0.1 ( I was testing a https connection over localhost)

This is how I did it using the python core libraries ssl and http.client (cf docs), as it seemed cleaner than the solutions I found online using the requests library.

    import http.client as http
    import ssl

    dst= 'sever.infsec.local' # dns record was added to OS
    src = ('127.0.0.2',0) # 0 -> select available port 

    context = ssl.SSLContext()
    context.load_default_certs() # loads OS certifcate context

    request = http.HTTPSConnection(dst, 443, context=context,
                                   source_address=src)

    request.connect()

    request.request("GET", '/', json.dumps(request_data))
    response = request.getresponse()
tlips
  • 121
  • 1
  • 4