Find a key inside a script tag with BeautifulSoup

Question

What I'm trying to do is acquire a product ID from a script tag inside an HTML document. Unfortunately, StockX doesn't offer a public API, so I have to scrape the data from an HTML document. Here are my attempts at it (both work):

Attempt 1

import requests

PRODUCT_URL = 'https://stockx.com/supreme-steiff-bear-heather-grey'
HEADERS = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36'}

response = requests.get(url=PRODUCT_URL, headers=HEADERS).text
PRODUCT_ID = response[response.find('"product":{"id":"')+17:].partition('"')[0]
PRODUCT_NAME = response[response.find('<title>')+7:].partition('<')[0]

Attempt 2

from bs4 import BeautifulSoup
import requests

# Gets HTML document
PRODUCT_URL = 'https://stockx.com/supreme-steiff-bear-heather-grey'
HEADERS = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36'}
html_content = requests.get(url=PRODUCT_URL, headers=HEADERS)

# Make BeautifulSoup parser from HTML document
soup = BeautifulSoup(html_content.text, 'html.parser')

# Get product name
PRODUCT_NAME = soup.title.text

# Get script tag data with product ID
js_content = soup.find_all('script', type='text/javascript')[9].text
PRODUCT_ID = js_content[50:86]

print(PRODUCT_ID)

Output: 884861d2-abe6-4b09-90ff-c8ad1967ac8c

However, I feel like there is a better approach to this problem instead of just "hard-coding" in where to find the ID.

If you view the page source of the product URL and do a search for "product":{"id":, you will find that the ID is inside a nested dictionary that is assigned to an object and inside a tag.

Is there any better way to go about obtaining the product ID from an HTML document?

EDIT: Here is the content of html_content: https://gist.github.com/leecharles50/9b6b11fb458767cabcfc0ed4f961984d

You should include a minimal example of `html_content.text` - [mcve]. — wwii, Nov 13 '19 at 00:05
OP gives you the URL and headers needed to get `html_content`, seems minimal _and_ reproducible to me — wpercy, Nov 13 '19 at 00:14
Maybe a regex like ```r'"product":{"id":"(\w{8}-(?:\w{4}-){3}\w{12}){1}"'```? — accdias, Nov 13 '19 at 00:37

score 1 · Accepted Answer · answered Nov 13 '19 at 00:32

My first idea was to parse the JavaScript inside the tag. There is a package called slimit that can do this. See for example this answer.

However, in your case there is an even easier solution. I searched the DOM for the id you gave (884861d2-abe6-4b09-90ff-c8ad1967ac8) and found an occurrence inside the following tag:

<script type="application/ld+json">
    {
        [...]
        "sku" : "884861d2-abe6-4b09-90ff-c8ad1967ac8c",
        [...]
    }
</script>

which contains valid JSON. Simply find the tag with BeautifulSoup:

tag = soup('script', {'type': 'application/ld+json'})[-1]

and decode the JSON within:

import json
product_id = json.loads(tag.text)['sku']

Ah nice, you beat me to it, although I believe this will fail on certain pages to some of the JSON contents. I'm currently working on getting that aspect figured out. — AMC, Nov 13 '19 at 00:35
This looks promising! Will try to implement this solution. :) — leecharles_, Nov 13 '19 at 00:57

AMC · Answer 2 · 2019-11-13T00:55:34.193

As you can see by the product URL, this has been tested on multiple product pages.

import requests
import json
from bs4 import BeautifulSoup

#product_url = 'https://stockx.com/supreme-steiff-bear-heather-grey'
product_url = 'https://stockx.com/air-jordan-1-retro-high-shattered-backboard-3'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36'}

html_content = requests.get(url=product_url, headers=headers)

soup = BeautifulSoup(html_content.text, 'lxml')

script_tags = soup.find_all('script', attrs={'type': 'application/ld+json'})

product_info_text = script_tags[-1].text

# contains a bunch of useful info
product_info_json = json.loads(product_info_text, strict=False)

print(json.dumps(product_info_json, indent=4))

product_sku = product_info_json['sku']

print(product_sku)

I will try to implement the use of a SoupStrainer.

I think you forgot a line before `product_info_json = ...` as `product_info_text` is undefined at that point. — Seb, Nov 13 '19 at 00:55
@Seb I thought I had already posted a comment in response, that’s so strange... Anyway, thank you for pointing that out, I probably never would have noticed otherwise! — AMC, Nov 15 '19 at 03:19

score 1 · Answer 3 · answered Nov 13 '19 at 00:59

Here is an alternative using regex:

import requests
import re

product_uuid = re.compile(r'"product":{"id":"(\w{8}-(?:\w{4}-){3}\w{12}){1}"')
product_name = re.compile(r'<title>(.*)</title>')

url = 'https://stockx.com/supreme-steiff-bear-heather-grey'
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36'
}

content = requests.get(url, headers=headers)

if content.ok:
    PRODUCT_NAME = product_name.findall(content.text)[0]
    PRODUCT_UUID = product_uuid.findall(content.text)[0]

    print(PRODUCT_NAME)
    print(PRODUCT_UUID)

Slightly hard-coded but easy to adjust and depends only on standard modules.

score 0 · Answer 4 · answered Jul 15 '22 at 09:34

0

If you want to scrape on large volumes, you can use the API of Piloterr

answered Jul 15 '22 at 09:34

Josselin

1

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Jul 16 '22 at 07:58
While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - [From Review](/review/late-answers/32254019) – Simas Joneliunas Jul 20 '22 at 02:33
Understood @SimasJoneliunas. I'll be careful next time. – Josselin Jul 21 '22 at 09:24

Find a key inside a script tag with BeautifulSoup

4 Answers4