1

I am developing a script that will eventually extract data from this website. I am receiving the "Insecure Request Warning" message. This is a script I will only use for myself and not for commercial use. How vulnerable am I to a man in the middle attack? I have little to no knowledge about SSL and don't want to put myself in danger while running this script. I am asking if it is safe for me to run the script on my own, or if I need to look into better security measures. Thank you for the help!

import requests

url = 'https://midwestauction.com/'    
response = requests.get(url, verify = False)    
print(response.status_code)
serverSentinel
  • 994
  • 6
  • 20
Zach Cook
  • 63
  • 1
  • 8

1 Answers1

2

requests won't automatically download extra certificates, but you can do this yourself, e.g. following suggestions in: https://stackoverflow.com/a/28667850/1358308

  1. ssllabs reports that we need "Go Daddy Secure Certificate Authority - G2" and includes SHA256 fingerprint 973a4127...
  2. searching for this points us to: https://ssl-ccp.godaddy.com/repository where we can download gdig2.crt.pem
  3. add this to the certifi PEM file, I did this with a combination of web browser, Python and shell scripts, but you can do it all in Python with:
import requests
import certifi

gdig2_url = 'https://ssl-ccp.godaddy.com/repository/gdig2.crt.pem'
local_pem = 'midwestauction.pem'

with open(certifi.where(), 'rb') as fd:
    pem = fd.read()

with requests.get(gdig2_url) as res:
    res.raise_for_status()
    pem += res.content

with open(local_pem, 'wb') as fd:
    fd.write(pem)

then you can use this in Python with:

requests.get(url, verify=local_pem)
Sam Mason
  • 15,216
  • 1
  • 41
  • 60
  • I just googled for the fingerprint, but searching for the name works as well. you want to make sure that you end up at a site that isn't likely to be "fake", i.e. it's actually associated with the certificate authority. doing this search once, manually, is likely to be relatively safe, but automating this starts to get difficult – Sam Mason Sep 05 '19 at 08:02
  • 1
    Another way to get the PEM file: Go to https://www.ssllabs.com/ssltest/analyze.html?d=midwestauction.com&latest. After it finishes scanning, click the "Click here to expand" button. On the right-hand side of the line that begins with "Path #1: Trusted" there is a [download link](https://www.ssllabs.com/ssltest/getTestTrustPath?d=midwestauction.com&cid=f72639a08632b880ae7b3adda1afb3e2919a9f98cc4b1d295ba2a0bca2a1aa81&time=1567684099987&id=1&trustStore=1) to a PEM file containing 3 certificates, including the missing one. – unutbu Sep 05 '19 at 12:01