0

I'm scraping a random web page with Scrapy. I created the project, but when I tried to run my spider, it couldn't import scrapy.

This is my spider

import scrapy
import logging

# Spider for truecar.com
class TruecarSpider(scrapy.Spider):
    name = "truecar"

    def start_requests(self):
        urls = ['https://www.truecar.com/used-cars-for-sale/listings/tesla/model-3/']

        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        all_listings = response.xpath('//div[@data-test="allVehicleListings"] > ul')

        for tesla in all_listings:
            aux = tesla.xpath('//div[@class="linkable card card-shadow vehicle-card"]')
            make_model = aux.xpath('@aria-label')#aux.css('::attr(aria-label)')
            year = make_model.xpath('@aria-label').get()
            model_raw = make_model.css('span.vehicle-header-make-model').get()
            model = model_raw[model_raw.find('>')+1:-7]. replace ('<! ----3"')
            tesla_data = {
                'url': 'http://truecar.com' + tesla.css('a::attr (href)').get(),
                'model': year + ' ' + model,
                'mileage': tesla.css('div[data-test="cardContent"] > div > div. text-truncate: :text').get(),
                'price': tesla.css('h4: :text').get(),
            }

            yield tesla_data

I have it installed through pip install scrapy so I tried through the terminal on VS Code to check if it exists.

This is what I got

screenshot of VS Code

I have python 3.10.1 and scrapy 2.7.1

Gino Mempin
  • 25,369
  • 29
  • 96
  • 135
Pedro Lemos
  • 1
  • 3
  • 2
    That image is showing python 3.7 and you say you have python 3.10, so I'm guessing you've got multiple python installs and scrapy is installed for the wrong one. To install to python 3.10, try `python-3.10 -m pip install scrapy` or `pip3.10 install scrapy` – MatBailie Jan 31 '23 at 23:32
  • https://stackoverflow.com/a/4910393/53341 – MatBailie Jan 31 '23 at 23:37
  • 1
    Actually, it looks like you installed to 3.10 but are trying to run it from 3.7, so try running python with `python3.10` instead of `python`. You'll also need to ensure your IDE (spider, vscode, whatever) is set up to use the correct python install. – MatBailie Jan 31 '23 at 23:42
  • How do I ensure that my IDE is set with the correct version? I think everything else is correct – Pedro Lemos Jan 31 '23 at 23:48
  • 2
    Check the manual? https://docs.spyder-ide.org/current/faq.html#using-existing-environment – MatBailie Jan 31 '23 at 23:50
  • And look at `venvs`, they will allow you to manage libraries more conveniently – Gameplay Feb 01 '23 at 10:14

1 Answers1

0

Thanks to @MatBailie's comments, it helped me to see that my python version on the IDE was wrong. I'm using VS Code and if any of you have the same problem you can check it here, on the blue bar below the terminal. If it doesn't appear, make sure you have Python extension and that it is reloaded:

enter image description here

Gino Mempin
  • 25,369
  • 29
  • 96
  • 135
Pedro Lemos
  • 1
  • 3