Replicate browser actions with a python script using Fiddler

Question

I am trying to log in to a website using python and the requests module.

My problem is that I am still seeing the log in page even after I have given my username / password and am trying to access pages after the log in - in other words, I am not getting past the log in page, even though it seems successful.

I am learning that it can be a different process with each website and so it's not obvious what I need to add to fix the problem.

It was suggested that I download a web traffic snooper like Fiddler and then try to replicate the actions with my python script.

I have downloaded Fiddler, but I'm a little out of my depth with how I find and replicate the actions that I need.

Any help would be gratefully received.

---

My original code:

import requests
payload = {
    'login_Email': 'xxxxx@gmail.com',
    'login_Password': 'xxxxx'
}

with requests.Session() as s:
    p = s.post('https://www.auction4cars.com/', data=payload)
    print p.text

Answer 1

If you look at the browser developer tools, you may see that the login POST request needs to be submitted to a different URL:

https://www.auction4cars.com/Home/UserLogin

Note that also the payload needs to be:

payload = {
    'login_Email_or_Username': 'xxxxx@gmail.com',
    'login_Password': 'xxxxx'
}

I'd still visit the login page before doing that and set the headers:

HOME_URL = 'https://www.auction4cars.com/'
LOGIN_URL = "https://www.auction4cars.com/Home/UserLogin"

with requests.Session() as s:
    s.headers = {
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
    }
    s.get(HOME_URL)

    p = s.post(LOGIN_URL, data=payload)
    print(p.text)  # or use p.json() as, I think, the response format is JSON