Stack Overflow
Stack Overflow

Questions tagged [grequests]

GRequests allows you to use Requests with Gevent to make asyncronous HTTP Requests easily.

GRequests allows you to use requests with gevent to make asynchronous HTTP Requests easily.

139 questions
43
votes
3 answers

In what way is grequests asynchronous?

I've been using the python requests library for some time, and recently had a need to make a request asynchronously, meaning I would like to send off the HTTP request, have my main thread continue to execute, and have a callback called when the…
cacois
  • 2,036
  • 2
  • 19
  • 19
28
votes
3 answers

to Json

I've never done any object oriented programming, only basic script writing. I'm playing around with grequests rs = (grequests.get('https://api.github.com/repositories?since='+str(page), auth=(login, password)) for page in pages) blah =…
Morgan Allen
  • 3,291
  • 8
  • 62
  • 86
24
votes
2 answers

Using grequests to make several thousand get requests to sourceforge, get "Max retries exceeded with url"

I am very new to all of this; I need to obtain data on several thousand sourceforge projects for a paper I am writing. The data is all freely available in json format at the url http://sourceforge.net/api/project/name/[project name]/json. I have a…
crf
  • 1,810
  • 3
  • 15
  • 23
19
votes
3 answers

Using Requests library to make asynchronous requests with Python 3.7

I need to make asynchronous requests using the Requests library. In Python 3.7 if I try from requests import async I get SyntaxError: invalid syntax. async has become a reserved with in Python 3.7. How to I get around this situation?
sigmus
  • 2,987
  • 3
  • 23
  • 31
18
votes
2 answers

In requests library, how can I avoid "HttpConnectionPool is full, discarding connection" warning?

I'm using python requests library with sessions: def _get_session(self): if not self.session: self.session = requests.Session() return self.session And sometimes I'm getting this warning in my logs: [2014/May/12 14:40:04 WARNING ]…
mnowotka
  • 16,430
  • 18
  • 88
  • 134
15
votes
1 answer

How do I make HTTP POST requests with grequests

I tried the following: import grequests urls = ['http://localhost/test', 'http://localhost/test'] params = {'a':'b', 'c':'d'} rs = (grequests.post(u, params) for u in urls) grequests.map(rs) But it says the following: File "search.py", line 6, in…
Crypto
  • 1,217
  • 3
  • 17
  • 33
12
votes
2 answers

Understanding requests versus grequests

I'm working with a process which is basically as follows: Take some list of urls. Get a Response object from each. Create a BeautifulSoup object from the text of each Response. Pull the text of a certain tag from that BeautifulSoup object. From my…
Brad Solomon
  • 38,521
  • 31
  • 149
  • 235
11
votes
2 answers

Celery + Eventlet + non blocking requests

I am using Python requests in celery workers to make large number of (~10/sec) API calls(includes GET,POST, PUT, DELETE). Each request takes around 5-10s to complete. I tried running celery workers in eventlet pool, with 1000 concurrency. Since…
Maddy
  • 791
  • 1
  • 8
  • 22
10
votes
2 answers

parallel post requests using multiprocessing and requests in Python

I have small code snippet as below: import requests import multiprocessing header = { 'X-Location': 'UNKNOWN', 'X-AppVersion': '2.20.0', 'X-UniqueId': '2397123', 'X-User-Locale': 'en', 'X-Platform': 'Android', 'X-AppId':…
qre0ct
  • 5,680
  • 10
  • 50
  • 86
9
votes
2 answers

GRequests monkey patch warning

I get the following warning each time , though the module works as expected : /usr/local/lib/python3.7/site-packages/grequests.py:21: MonkeyPatchWarning: Monkey-patching ssl after ssl has already been imported may lead to errors, including…
Ram K
  • 1,746
  • 2
  • 14
  • 23
9
votes
2 answers

Strange blocking behaviour with gevent/grequests over HTTPS

The following code sends a request every 200ms and should handle the responses asynchronously whenever they come. Over HTTP it works as expected - a request is sent every 200ms and the response callback is called independently whenever a response…
akxlr
  • 1,142
  • 9
  • 23
8
votes
2 answers

grequests with requests has collision

I am using grequests python module to call some APIs. I want to make two functions. A single request(use requests module) A multiple request(use grequests module) When I use two modules in two different files, it runs normally, but when I import…
Shaffron
  • 97
  • 1
  • 3
  • 11
7
votes
1 answer

imap vs. map in grequests library

As far as I know the difference between map and imap is that map waits for all requests to finish and then returns the ordered data. Whereas imap returns data immediately and order less. When I use: urls = [...some_data...] rs = (grequests.get(u,,…
max
  • 677
  • 1
  • 9
  • 34
7
votes
2 answers

Unable to import grequests for AWS Lambda

I'm running an AWS Lambda script with a Python 2.7 runtime. However, whenever it initializes it begins to import the grequests library but fails because of it's dependency on gevent: Gevent is required for grequests. It seems it is successfully…
Tyler Mills
  • 353
  • 1
  • 3
  • 15
7
votes
3 answers

grequests pool with multiple request.session?

I want to make a lot of url requets to a REST webserivce. Typically between 75-90k. However, I need to throttle the number of concurrent connections to the webservice. I started playing around with grequests in the following manner, but quickly…
Marcel Wilson
  • 3,842
  • 1
  • 26
  • 55
1
2 3
…
9 10