How manage multiple proxies in a thread environment?

Question

I want each thread to run with a different IP. is it possible to use a different ip for each thread? So far i can't assign a different IP address to each thread. The goal is to do web scraping without being banned.

f1-> ip X

f2-> ip y

f3-> ip z

Here is my code:

import requests
from stem.control import Controller
from stem import Signal
import json
import time  
import concurrent.futures  
from fake_useragent import UserAgent



def get_tor_session():
    # initialize a requests Session
    session = requests.Session()
    # setting the proxy of both http & https to the localhost:9050 
    # this requires a running Tor service in your machine and listening on port 9050 (by default)
    session.proxies = {"http": "socks5h://localhost:9150", "https": "socks5h://localhost:9150"}
    return session

def renew_connection():
    with Controller.from_port(port=9051) as c:
        c.authenticate()
        # send NEWNYM signal to establish a new clean connection through the Tor network
        c.signal(Signal.NEWNYM)

ua=UserAgent()
headers = {'User-Agent':ua.random}
urlA = 'https://api.ipify.org'

def serveur():
    renew_connection()
    s = get_tor_session()
    rA = s.get(urlA, headers=headers)
    print(rA.text)

def Th():
    with concurrent.futures.ThreadPoolExecutor() as executor:
        f1 = executor.submit(serveur)
        f2 = executor.submit(serveur)
        f3 = executor.submit(serveur)

Th()

here is the current return:

185.220.101.14
185.220.101.14
185.220.101.14

Does this answer your question? [How i can get new ip from tor every requests in threads?](https://stackoverflow.com/questions/56733397/how-i-can-get-new-ip-from-tor-every-requests-in-threads) — drew010, Nov 01 '20 at 20:27

score 0 · Answer 1 · answered Nov 01 '20 at 15:03

It could be useful:

I had a task to implement auto-updating multi proxy.

I used Tor for it. Here is a small package, how to run multiple Tor instances: https://github.com/detonavomek/tor-pool

I use macOS. There is also privoxy helper for proxies. Here is a privoxy Python package: https://github.com/detonavomek/privoxy-pool

Information about how to use it is here: https://github.com/detonavomek/privoxy-pool/blob/master/test_controller.py

To install:

pip install git+https://github.com/detonavomek/privoxy-pool.git@https://github.com/detonavomek/privoxy-pool.git

How manage multiple proxies in a thread environment?

1 Answers1