3

Let us say I am sending a GET request to an URL which's DNS record contains multiple IP addresses (DNS Round-Robin). For example, to the https://www.twitter.com. If we do dig of it, we see:

;; ANSWER SECTION:
twitter.com.        60  IN  A   104.244.42.193
twitter.com.        60  IN  A   104.244.42.129

I have tried to use DEBUG logging as described here: https://stackoverflow.com/a/16630836/1479414 however it does not show which IP address it uses.

Furthermore, I looked also at this answer: https://stackoverflow.com/a/22513161/1479414 about how to see the destination IP address, but it is not what I actually want to see -- this one shows me the finally used IP address.

I want rather to see which IP addresses are tried during all attempts, while the library performs DNS failover. Something like:

Trying to connect to 104.244.42.193... failed, trying to connect using another IP address

Is it possible to see somehow?

Andremoniy
  • 34,031
  • 20
  • 135
  • 241
  • [This is the line that resolves name to IP address](https://github.com/urllib3/urllib3/blob/11d68efa7c150823472f0e5309c3a08a1a10c2f2/src/urllib3/util/connection.py#L61). [`socket.getaddrinfo()`](https://docs.python.org/3/library/socket.html#socket.getaddrinfo) produces a list of tuples to try. – Martijn Pieters Jan 12 '20 at 09:40
  • @MartijnPieters okay, so do you mean that's not possible to log somehow? – Andremoniy Jan 12 '20 at 09:45
  • 1
    There is zero logging there, so you'd have to patch that area. – Martijn Pieters Jan 12 '20 at 09:46
  • Thanks @MartijnPieters, it sounds like a definite answer which I would accept for sure. – Andremoniy Jan 12 '20 at 09:47

1 Answers1

3

The urllib3 library is responsible for this, in the urllib3.util.connection.create_connection() function. There's a loop there that checks each result from socket.getaddrinfo():

for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
    #  ....
    try:
        sock = socket.socket(af, socktype, proto)
        # ...
        sock.connect(sa)
        return sock

    except socket.error as e:
        # ...

Unfortunately, there is no logging used in this section. If you must have logging, you'd have to copy the function source, add in the logging you need, then monkeypatch the urllib3 code to use your version instead.

The only location that uses that function uses it as an attribute on the imported module, so at least monkeypatching is as simple as assigning it back to original function location:

import urllib3.util.connection

def create_connection_with_logging(
    address,
    timeout=socket._GLOBAL_DEFAULT_TIMEOUT,
    source_address=None,
    socket_options=None,
):
    # ....

urllib3.util.connection.create_connection = create_connection_with_logging
Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343