Scrapy Extract ld+JSON

Question

How to extract the name and url?

quotes_spiders.py

import scrapy
import json

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ["http://www.lazada.com.my/shop-power-banks2/?price=1572-1572"]

    def parse(self, response):
        data = json.loads(response.xpath('//script[@type="application/ld+json"]//text()').extract_first())
        //how to extract the name and url?
        yield data

Data to Extract

<script type="application/ld+json">{"@context":"https://schema.org","@type":"ItemList","itemListElement":[{"@type":"Product","image":"http://my-live-02.slatic.net/p/2/test-product-0601-7378-08684315-8be741b9107b9ace2f2fe68d9c9fd61a-webp-catalog_233.jpg","name":"test product 0601","offers":{"@type":"Offer","availability":"https://schema.org/InStock","price":"99999.00","priceCurrency":"RM"},"url":"http://www.lazada.com.my/test-product-0601-51348680.html?ff=1"}]}</script>

What happens with your current code? Is it different from what you expect? Can you show what you get and what is missing? — paul trmbrth, Jul 06 '17 at 07:49

score 9 · Answer 1 · answered Jan 06 '18 at 21:04

This line of code returns a dictionary with the data you want:

data = json.loads(response.xpath('//script[@type="application/ld+json"]//text()').extract_first())

All you need to do is to access it like:

name = data['itemListElement'][0]['name']
url = data['itemListElement'][0]['url']

Given that the microdata contains a list you will need to check you are referring to the correct product in the list.

score 4 · Answer 2 · answered Aug 13 '17 at 15:30

4

A really easy solution for this would be to use https://github.com/scrapinghub/extruct. It handles all the hard parts of extracting structured data.

answered Aug 13 '17 at 15:30

Thomas Nys

359
3
16

Scrapy Extract ld+JSON

2 Answers2

Linked