Scrapy playwright (python) giving 403 in headless = True and 200 in headless = false?

Question

I am Scraping with scrapy-playwright an ecommerce site where when I scrap with headless: True, I am getting 403 error but, with Headless False I am getting 200,I even tried randomizing User agent still getting blocked.

The scrap is running with firefox playwright driver annd webkit driver but, its taking so much time, I want to run it with chromium

 def make_request_from_data(self, data):
    payload = json.loads(data)
    isbn = payload["isbn"]

    url = f"https://www.barnesandnoble.com/s/{isbn}"
    meta = {
        "region": self.region,
        "isbn": isbn,
        "playwright": True,
        "playwright_include_page": True,
        "playwright_context": f"context-{isbn}",
        "playwright_context_kwargs": {
            "java_script_enabled": True,
        },
    }
    headers = {
    "accept-encoding": "gzip, deflate, br",
    "accept-language": "en",
    "cache-control": "no-cache",
    "pragma": "no-cache",
    "sec-fetch-dest": "document",
    "sec-fetch-mode": "navigate",
    "sec-fetch-site": "none",
    "sec-fetch-user": "?1",
    "upgrade-insecure-requests": "1",
    "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
    }  
    yield Request(
        headers = headers,
        url = url, 
        callback=self.parse,
        errback=self.close_context_on_error,
        meta=meta,
        dont_filter=True,
    )

Isbn is the book code, and my wild guess is with the chromium version, I dont know how to downgrade chromium version in playwright

Url I want to scrap `https://www.barnesandnoble.com/w/harry-potter-and-the-chamber-of-secrets-j-k-rowling/1004338523?ean=9781781100509` — aditya narayanan, Jan 11 '23 at 14:53
Please provide enough code so others can better understand or reproduce the problem. — Community, Jan 11 '23 at 15:10

score 0 · Answer 1 · answered Apr 27 '23 at 21:53

I just fixed the same issue by adding a custom UserAgent on the browser context This links helped a lot.

https://jsoverson.medium.com/how-to-bypass-access-denied-pages-with-headless-chrome-87ddd5f3413c

How to add custom headers in Playwright

[OneTimeSetUp]
public async Task OneTimeSetUp()
{
    // Set up Playwright
    _playwright = await Playwright.CreateAsync();
    var browserType = _playwright.Chromium;
    _browser = await browserType.LaunchAsync(new BrowserTypeLaunchOptions
    {
        Headless = true
    });
    var browserContext = await _browser.NewContextAsync(new BrowserNewContextOptions
    {
        BaseURL = baseURL,
        UserAgent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36"
    });
    _page = await browserContext.NewPageAsync();
    await _page.SetViewportSizeAsync(1366, 768 );
}

Scrapy playwright (python) giving 403 in headless = True and 200 in headless = false?

1 Answers1