Attribute Error when using Python requests to scrape a website

Question

import requests
import json

url_oc = "https://tradeguide.in/future_live"
url = f"https://tradeguide.in/future_live/get_data?symbol=NIFTY&expiry=2021-05-27&interval=5"
headers = {'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36',
           'accept-language: en-GB,en-US;q=0.9,en;q=0.8', 'accept-encoding: gzip, deflate, br'}
session = requests.Session()
request = session.get(url_oc, headers=headers, timeout=5)
cookies = dict(request.cookies)
response = session.get(url, headers=headers, timeout=5, cookies=cookies)
r=response.json()

I am unfamiliar with scraping websites using Python. I tried running this code, but I am getting the following error:

AttributeError: 'set' object has no attribute 'items'.

I am trying to retrieve data futures open interest using this of all the stocks.

You might want to check https://stackoverflow.com/questions/6260457/using-headers-with-the-python-requests-librarys-get-method on how to set headers for the `requests` library. — Progman, May 01 '21 at 09:30

score 0 · Answer 1 · answered May 01 '21 at 10:55

The headers parameter takes a dictionary, not a set. You must split the header key and value into their own seperate parts, like so:

headers = {
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36',
    'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8',
    'accept-encoding': 'gzip, deflate, br'
}

Attribute Error when using Python requests to scrape a website

1 Answers1