1

I am using

$ python --version                                                                                                                                                                      
Python 3.7.3

I am trying the following code to log the output of request to a file

import requests
import logging

import http.client as http_client
http_client.HTTPConnection.debuglevel = 1

# You must initialize logging, otherwise you'll not see debug output.
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

handler = logging.FileHandler('test2.log',mode='a')
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
handler.setLevel(logging.DEBUG)  

requests_log.addHandler(handler)

requests.get('https://httpbin.org/headers')

What i see in the test2.log is

2021-03-07 17:50:22,312 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): httpbin.org:443
2021-03-07 17:50:22,376 - urllib3.connectionpool - DEBUG - https://httpbin.org:443 "GET /headers HTTP/1.1" 200 225

Why am I not able to see the other output which shows in the console in the log, i.e as below

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): httpbin.org:443
send: b'GET /headers HTTP/1.1\r\nHost: httpbin.org\r\nUser-Agent: python-requests/2.24.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Sun, 07 Mar 2021 12:20:21 GMT
header: Content-Type: application/json
header: Content-Length: 225
header: Connection: keep-alive
header: Server: gunicorn/19.9.0
header: Access-Control-Allow-Origin: *
header: Access-Control-Allow-Credentials: true
DEBUG:urllib3.connectionpool:https://httpbin.org:443 "GET /headers HTTP/1.1" 200 225

HOw to get the headers data also into the logs

Santhosh
  • 9,965
  • 20
  • 103
  • 243

1 Answers1

0

Have you tried to monkey-patch the print function of http.client module? Unfortunately, it uses prints instead of Python logging module, hence this behaviour.

Adding add the following piece of code before your request should work:

def print_to_log(*args):
    requests_log.debug(" ".join(args)) 
http_client.print = print_to_log

Related answers (Log all requests from the python-requests module and Python HTTP request and debug level logging to the log file).

Matt-Mac-Muffin
  • 177
  • 8