Python Redis Multiprocessing - redis.exceptions.ConnectionError -

Question

I have written a program that does multiprocessing and does Scan, Get, and Set to Redis. The reason we went with Redis is to speed it up.

Does anybody have any recommendations. I tried putting a retry loop around the get statements (and can do that for set as well), but the Scan statements will be more tricky. And is there some parameter I scan change or increase to avoid this?

Error:

redis.exceptions.ConnectionError: Error 10048 connecting to 192.168.5.51:6379. Only one usage of each socket address (protocol/network address/port) is normally permitted.
  File "C:\Apps\ProcessData\redis\connection.py", line 1192, in get_connection
    connection.connect()
  File "C:\Apps\ProcessData\redis\connection.py", line 563, in connect
    raise ConnectionError(self._error_message(e))
  File "C:\Apps\ProcessData\redis\connection.py", line 563, in connect
    raise ConnectionError(self._error_message(e))

I think part two of the error was this: waiting for it to happen again to get the exact text:

Error 10048 connecting to 192.168.5.51:6379. Only one usage of each socket address (protocol/network address/port) is normally permitted.

For the code that does a redis-get, I am trying the following:

redis_max_retries = 72
redis_retries = 0
redis_get_success = False
sleep_time_in_seconds = .1
while not redis_get_success and redis_retries < redis_max_retries:
    try:
       json_str = redis_obj.get(redis_key)
       redis_get_success = True
    except redis.exceptions.ConnectionError as e:
        redis_retries += 1
        time.sleep(sleep_time_in_seconds)

if not redis_get_success:
    print("redis_retries=", redis_retries, " on json_str = redis_obj.get(redis_key) in process_intervals" )
    sys.exit(0)

But I also get the error on a scan, which look trickier to retry.

    for keybatch in batcher(redis_obj.scan_iter(key_pattern), 1500):
        batch_counter += 1
        # print(batch_counter, "keybatch=", keybatch)

        for key in keybatch:
            if key is not None:
                #print("Trying key=", key)
                matching_keys_found += 1
                etc...

score 0 · Answer 1 · answered Oct 29 '20 at 18:58

0

Try to connect to online Redis hosting site like RedisLabs.com and connect using documentation or you can try changing the port. Error is either arising from port or misconfigured Redis server setup. Also for the same port and address see Only one usage of each socket address (protocol/network address/port) is normally permitted?

answered Oct 29 '20 at 18:58

Ved Nig

59
11

1

We moved to self-hosted Redis because of the memory we needed. It also happens in other programs, so not related to multiprocessing. Still trying to figure out if my retry works, maybe I need to increase retry time. I was just reading this: https://help.socketlabs.com/docs/how-to-fix-error-only-one-usage-of-each-socket-address-protocolnetwork-addressport-is-normally-permitted – NealWalters Oct 29 '20 at 22:19
I think I saw that post, but we are hosting on Ubuntu, and it didn't mention Redis at all. Was hoping that redis_py library might solve, since I'm not coding any sockets directly. But maybe we do need to open some more sockets/ports on that server... Also it normally works fine when running one Python program at a time, or no multiprocessing. So it's not a documentation/port issue. I am able to connect, I think I'm out of connections. – NealWalters Oct 29 '20 at 22:22
Try using connection_pool property and disconnect on completion. This might pave way for more connections. Unless it is important to have more than one connections simultaneously. – Ved Nig Oct 30 '20 at 04:55
Have you seen this: https://redis.io/topics/clients – Ved Nig Oct 30 '20 at 04:58

Python Redis Multiprocessing - redis.exceptions.ConnectionError -

1 Answers1