Crawl website with scrapy within application

Question

I am developing an application to crawl websites using Scrapy .I had a problem I want run the scrapy crawl website command within the application. Because the crawler will run from the browser.

Any Ideas ??

score 1 · Accepted Answer · edited May 23 '17 at 12:23

1

You should call a command from within your Python code. See here how that's done: Calling an external command in Python

edited May 23 '17 at 12:23

Community

1
1

answered Nov 04 '15 at 09:59

user1797792

1,169
10
26

And, did it work? Then please mark this as an answer :) – user1797792 Nov 04 '15 at 13:23
Though I agree with @user1797792 and have done some similar things, just wanted to know what web framework you are using? I like Flask, but that means that you want to do stuff from the ground up, while Django has tons of apps, like this one [Django dynamic scraper](https://django-dynamic-scraper.readthedocs.org/en/latest/) and probably more. – dasdachs Nov 04 '15 at 15:27
I use Django . But i use scrappy for scrapping – user3383301 Nov 04 '15 at 17:55

Crawl website with scrapy within application

1 Answers1