What does it mean when it says:
requests.exceptions.ConnectionError: None: Max retries exceeded with url: /myurl (Caused by None)
Specifically what does "Caused by None" mean?
I have a python client and a simple clojure server running on the same machine. The server runs on compojure+http-kit with 4 threads. The client continuously submit POST requests using 3 to 4 processes using multiprocessing.Pool with 3 worker processes.
Every so often, the client would die with the ConnectionError described above. I have settings retries=3 on the client side and increasing the queue size on the server to 1000000 with no effect.
Any help would be appreciated.
edit: A correction, I am actually sending POST requests not GET.
The actual script is too large to post here but basically it works like this. I have a function that calls post with some data:
def post_func(my_data, config):
return requests.post(my_url, data=json.dumps({"data": my_data, "config": config}))
A class that wraps multiprocessing.Pool:
class Executor(object):
def __init__(self, nprocs):
self.pool = Pool(processes=nprocs)
def execute(self, func, list_of_data):
return self.pool.map(func, list_of_data)
Another function that calls Executor.execute() with different configurations:
function eval(executor, list_of_data, config):
start = timer()
func = partial(post_func, config=config)
results = executor.execute(func, list_of_data)
taken = timer()-start
return results
A single Executor is reused for all eval() calls. eval() is then wrapped in a scoring function and given to pyswarm to optimise:
pso(score_func, lbs, ubs, swarmsize=20, maxiter=20, debug=True)
edit: something I probably should have done earlier, but catching the ConnectionError properly gives me this:
ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /my_url (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f0ccb4b5550>: Failed to establish a new connection: [Errno 99] Cannot assign requested address',))
I have now rewrote the script to reuse a single requests.Session, will know soon if this fixes the problem.
