0

I've installed scrapy and created a spider that works when run from the command line with the command scrapy crawl getBUCPower. My issue is that I need to run the spider from another script when something specific happens. I have a separate python script thats just a test for now, but it tries to run the spider by the line:

execfile("../scrapy/data_spider/data_spider/spiders/getBUCPower.py")

When I run that script, I immediately get the error:

File "getBUCPower.py", line 2, in
import scrapy
ImportError: No module named scrapy

I have scrapy installed correctly as it work when I run the scrapy crawl command so I am unaware of what the issue is right now.

Here is my spider

import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy import Spider
from scrapy.loader import ItemLoader

class DataSpider(Spider):
#sets the name of the spider
name = 'getBUCPower'
#sets the pipeline the spider should use
custom_settings = {
    'ITEM_PIPELINES': {
        'data_spider.pipelines.CsvPipeline': 300
    }
}
#sets the url the spider should crawl
allowed_domains = [ip]
start_urls = [ipHTTP]


def parse(self, response):
    #sets the information the sipder should grab
    BUCPower = response.xpath('//*[@id="mmwpaTxPower"]/text()').extract_first()
    #returns the information
    yield{"BUCPower" : BUCPower}

process = CrawlerProcess(settings={
'FEED_FORMAT': 'json',
'FEED_URI': 'items.json'
})

process.crawl(DataSpider)
process.start()

I currently have scrapy 1.6 and am using Python 2.7.16

The expected result is to run the spider from another script, but instead it has an issue importing scrapy.

ejderuby
  • 710
  • 5
  • 21
Strange
  • 1
  • 1
  • 2
  • How are you running the python script that contains `execfile()`? What version of Python is used? – John Gordon Aug 01 '19 at 19:44
  • currently i am just running it from command line by "python callGetBucPower.py" and i am using 2.7.16 – Strange Aug 01 '19 at 19:50
  • What output do you get from `python --version`? What is your `PYTHONPATH`? – John Gordon Aug 01 '19 at 19:57
  • Python 2.7.16 :: Anaconda, Inc. im not sure how to find my PYTHONPATH – Strange Aug 01 '19 at 19:59
  • You're using Anaconda? How did you install scrapy? – John Gordon Aug 01 '19 at 20:03
  • is this how you get the path? C:\Users\astanecek>python -c "import sys; print(sys.path)" ['', 'C:\\Users\\astanecek\\AppData\\Local\\Continuum\\anaconda2\\python27.zip', 'C:\\Users\\astanecek\\AppData\\Local\\Continuum\\anaconda2\\DLLs', 'C:\\Users\\astanecek\\AppData\\Local\\Continuum\\anaconda2\\lib', 'C:\\Users\\astanecek\\AppData\\Local\\Continuum\\anaconda2\\lib\\plat-win', 'C:\\Users\\astanecek\\AppData\\Local\\Continuum\\anaconda2\\lib\\lib-tk', 'C:\\Users\\astanecek\\AppData\\Local\\Continuum\\anaconda2', 'C:\\Users\\astanecek\\AppData\\Local\\Continuum\\anaconda2\\lib\\site-packages – Strange Aug 01 '19 at 20:03
  • pip install scrapy – Strange Aug 01 '19 at 20:03
  • I'm not sure pip is the right way to install packages for use with Anaconda. – John Gordon Aug 01 '19 at 20:04
  • how should i go about installing it? – Strange Aug 01 '19 at 20:07
  • I believe the command is `conda` instead of `pip`. – John Gordon Aug 01 '19 at 20:09
  • conda is not recognized as an internal or external command – Strange Aug 01 '19 at 20:09
  • check this https://stackoverflow.com/a/51996934/4949165 – Arun Augustine Aug 02 '19 at 04:17

3 Answers3

0

run anaconda prompt as administrator and then simply install scrapy by command

pip install scrapy
Samsul Islam
  • 2,581
  • 2
  • 17
  • 23
Tauqeer Sajid
  • 101
  • 1
  • 1
  • 10
0

Write the following code in anaconda/miniconda prompt:

conda install scrapy
h3t1
  • 1,126
  • 2
  • 18
  • 29
Kashif Alamdar
  • 13
  • 4
0

Installing protego package solved the issue for me.

conda install protego or pip install protego

MavenSanK
  • 1
  • 1