4

I have a site that I want to extract data from. The data retrieval is very straight forward.

It takes the parameters using HTTP POST and returns a JSON object. So, I have a list of queries that I want to do and then repeat at certain intervals to update a database. Is scrapy suitable for this or should I be using something else?

I don't actually need to follow links but I do need to send multiple requests at the same time.

Crypto
  • 1,217
  • 3
  • 17
  • 33

3 Answers3

8

How does looks like the POST request? There are many variations, like simple query parameters (?a=1&b=2), form-like payload (the body contains a=1&b=2), or any other kind of payload (the body contains a string in some format, like json or xml).

In scrapy is fairly straightforward to make POST requests, see: http://doc.scrapy.org/en/latest/topics/request-response.html#request-usage-examples

For example, you may need something like this:

    # Warning: take care of the undefined variables and modules!

    def start_requests(self):
        payload = {"a": 1, "b": 2}
        yield Request(url, self.parse_data, method="POST", body=urllib.urlencode(payload))

    def parse_data(self, response):
        # do stuff with data...
        data = json.loads(response.body)
R. Max
  • 6,624
  • 1
  • 27
  • 34
0

For handling requests and retrieving response, scrapy is more than enough. And to parse JSON, just use the json module in the standard library:

import json

data = ...
json_data = json.loads(data)

Hope this helps!

aIKid
  • 26,968
  • 4
  • 39
  • 65
  • I know how to handle the JSON responses but doesn't scrapy require start_urls and rules to be set in order to set up my callbacks? – Crypto Jan 09 '14 at 11:56
0

Based on my understanding of the question, you just want to fetch/scrape data from a web page at certain intervals. Scrapy is generally used for crawling.

If you just want to make http post requests you might consider using the python requests library.

Harry
  • 394
  • 2
  • 10
  • I want it to fetch multiple datapoints simultaneously and then wait for a certain interval before continuing. That's why I thought I'll use scrapy because it supports that. If I use the requests module then I'll have to do them one at a time. – Crypto Jan 09 '14 at 14:04
  • I mean asynchronously. – Crypto Jan 09 '14 at 15:26
  • I'm thinking if it's possible to set up Requests from the CrawlSpider class. – Crypto Jan 10 '14 at 08:10
  • I'm still not sure why you need to use scrapy. It is used to crawl over the web graph and scrape content but you dont seem to be doing that. Async requests are possible using the [grequests](https://github.com/kennethreitz/grequests) library. Look at [this](http://stackoverflow.com/q/9110593/1576996) question as well. – Harry Jan 10 '14 at 09:15