Traverse table rows with Scrapy

Question

Is it possible to scan a row from the table and vary only the column, to get a series of numbers. I am currently using many variables.

Example:

response.xpath('/html/body/div/table/tr[6]/td[counter in range 2 - 9]/p/span/text()').extract()

Code:

class MainSpider(scrapy.Spider):
    name = "main-spider"
    start_urls = ['http://www.institutosantatereza.com.br/boletins/turma_3_ano_ensino_medio/1652.htm']


    def parse(self, response):
        nome = response.xpath('/html/body/div/table/tr[2]/td[2]/p/b/span/text()').extract()
        serie = response.xpath('/html/body/div/table/tr[2]/td[7]/p/b/span/text()').extract()


        portugues1 = response.xpath('/html/body/div/table/tr[6]/td[2]/p/span/text()').extract()
        portugues2 = response.xpath('/html/body/div/table/tr[6]/td[3]/p/span/text()').extract()
        portuguesMedia1 = response.xpath('/html/body/div/table/tr[6]/td[4]/p/span/text()').extract()


        yield{
            "nome": nome[0],
            "serie": serie[0],
            "url": response.url,
            "disciplinas":{
                "portugues":{
                    'nota1': portugues1[0],
                    'nota2': portugues2[0],
                    'media1': portuguesMedia1[0], 
                }
            }
        }

if you want to get more than one element then instead of `[6]` you could get list with all rows, slice it in Python `all_rows[2:9]` and later use `for`-loop to work with every row separatelly - to get text in column. — furas, Nov 04 '19 at 02:28
see also [What is the XPath to select a range of nodes](https://stackoverflow.com/questions/3354987/what-is-the-xpath-to-select-a-range-of-nodes) — furas, Nov 04 '19 at 02:32

score 1 · Answer 1 · answered Nov 04 '19 at 04:18

There is no need to use many variables:

yield{
    "nome": nome[0],
    "serie": serie[0],
    "url": response.url,
    "disciplinas":{
        "portugues":{
            'nota1': response.xpath('/html/body/div/table/tr[6]/td[2]/p/span/text()').extract_first(), # or .get()
            'nota2': response.xpath('/html/body/div/table/tr[6]/td[3]/p/span/text()').get(), # or .extract_first()
        }
    }
}

Traverse table rows with Scrapy

1 Answers1