Unable to use session within a classmethod of a web-scraper

Question

I've created a python script using classmethod to fetch the profilename after loging in inputting the credentials in a webpage. The script is able to fetch the profilename in the right way. What I wish to do now is use session within classmethod. The session has already been defined within __init__() method. I would like to keep the existing design intact.

This is what I've tried so far:

import requests
from bs4 import BeautifulSoup

class StackOverflow:

    SEARCH_URL = "https://stackoverflow.com/users/login?ssrc=head&returnurl=https%3a%2f%2fstackoverflow.com%2f"

    def __init__(self,session):
        self.session = session

    @classmethod
    def crawl(cls,email,password):
        page = requests.get(cls.SEARCH_URL,headers={"User-Agent":"Mozilla/5.0"})
        sauce = BeautifulSoup(page.text, "lxml")
        fkey = sauce.select_one("[name='fkey']")["value"]
        payload = {"fkey": fkey,"email": email,"password": password,}
        res = requests.post(cls.SEARCH_URL,data=payload,headers={"User-Agent":"Mozilla/5.0"})
        soup = BeautifulSoup(res.text, "lxml")
        user = soup.select_one("div[class^='gravatar-wrapper-']").get("title")
        yield user

if __name__ == '__main__':
    with requests.Session() as s:
        result = StackOverflow(s)
        for item in result.crawl("email", "password"):
            print(item)

How can I use session taking from __init__ within classmethod?

This answer might help you .. https://stackoverflow.com/questions/12737740/python-requests-and-persistent-sessions — 0xPrateek, Jun 15 '19 at 20:00
I didn't create this post to learn how session works. I've a little knowledge on that. I wish to know how can I call session within classmethod fron init method. Thanks. — robots.txt, Jun 15 '19 at 20:22
You can't do it from a class method - it doesn't have access to `self` — Faboor, Jun 15 '19 at 22:47

score 1 · Accepted Answer · answered Jun 15 '19 at 23:05

You can't access self.session from a class method. Method __init__ is called when an instance of the class is created, however class methods are not bound to any particular instance of the class, but the class itself - that's why the first parameter is usually cls and not self.

You decided to create the session in the __init__, so it can be assumed that

so1 = StackOverflow()
so2 = StackOverflow()

keep their sessions separate. If that is indeed your intention, the crawl method should not be annotated with @classmethod. If you have crawl(self, email, pass): then you will still be able to use StackOverflow.SEARCH_URL and self.__class__.SEARCH_URL to get the value defined in StackOverflow class, or self.SEARCH_URL which will by default get the same value, but could be changed with so1.SEARCH_URL = "sth else" (but so2.SEARCH_URL would keep it's original value)

Unable to use session within a classmethod of a web-scraper

1 Answers1