Web scraping with log in for Lending club

Question

I am trying to scrape the current loan note status from the url column of the lending club download data. For example https://lendingclub.com/browse/loanDetail.action?loan_id=104046830 and it requires log in to extract info.

I've followed the steps to create log in session but it seems it cannot perform log in successfully. Result does not contain the right code. Can someone help me to identify the issue?

USERNAME = "username"
PASSWORD = "password"

LOGIN_URL = "https://www.lendingclub.com/auth/login?"

loan_id=96490539

URL = "https://lendingclub.com/browse/loanDetail.action?loan_id=96490539"

def main():
    session_requests = requests.session()

    # Get login csrf token
    result = session_requests.get(LOGIN_URL)
    tree = html.fromstring(result.text)
    authenticity_token = tree.xpath("//meta[@name='csrf-token']/@content")[0]

    # Create payload
    payload = {
        "login_email": USERNAME, 
        "login_password": PASSWORD, 
        "csrf-token": authenticity_token
    }

    # Perform login
    result = session_requests.post(LOGIN_URL, data = payload, headers = dict(referer = LOGIN_URL))

    # Scrape url
    result = session_requests.get(URL, headers = dict(referer = URL))
    return result

Are you tried to create a cookie, like: session = requests.session(config={'verbose': sys.stderr} — Fady Saad, Sep 13 '17 at 00:22
thanks! but i got ''session() got an unexpected keyword argument 'config''' issue i guess cuz im using python3 — Shirley, Sep 13 '17 at 00:44
you can write it in this way: s.config['verbose'] = sys.stderr — Fady Saad, Sep 13 '17 at 00:48

score 0 · Accepted Answer · answered Sep 13 '17 at 21:27

Although it looks weird what I'm suggesting, you might give this a try. According to chrome dev tools, it should suffice to fetch you a valid response.

import requests
from lxml import html

USERNAME = "username"
PASSWORD = "password"

LOGIN_URL = "https://www.lendingclub.com/account/login.action"

def main():

    payload={
    'login_url':'/browse/loanDetail.action?loan_id=96490539',
    'login_email':USERNAME,
    'login_password':PASSWORD,
    'offeredNotListedPromotionFlag':''
    }
    with requests.session() as session:
        session.headers={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36'}
        result = session.post(LOGIN_URL, data=payload, 
            headers={'Referer':'https://www.lendingclub.com/browse/loanDetail.action?loan_id=96490539','Content-Type': 'application/x-www-form-urlencoded'})

        return result

Web scraping with log in for Lending club

1 Answers1