Python requests module, how to issue multiple requests in the for loop?

Question

I wonder why when I call requests.get() method consequentially, like this:

response = requests.get(url.format("set"))
print(response.status_code)
response = requests.get(url.format("map"))
print(response.status_code)
response = requests.get(url.format("list"))
print(response.status_code)
response = requests.get(url.format("vector"))
print(response.status_code)
response = requests.get(url.format("string"))
print(response.status_code)

I got OK status for all requests, but when I do it in the for loop, like:

for word in fIn :
        response = requests.get(url.format(word))
        if(response.status_code == 200):
            print "OK"
        else:
            print(response.status_code)
            print "Error"
            print word

I got 400(Error) for all requests except the last one.

Additional info: there is related question on SO, where are mentioned 2 ways of coping with this situation: wait, headers.
wait doesn't work in my situation
and about headers - I don't know what to provide there.

Update: specific version, that I am trying to implement:

from lxml import html

import requests

fOut = open("descriptions.txt","w")

with open('dummyWords.txt') as fIn:
    for word in fIn :
        print word
        response = requests.get(url.format(word))
        if(response.status_code == 200):
            print "OK"
        else:
            print(response.status_code)
            print(word)

@Tim I am aware of rate limit, I don't know whether they have one. But it seems strange for me that typing requests manually works, while using for loop not. Maybe in the for loop every time new object for session is created and new session is established, while in manual requests one session is used? — spin_eight, Jul 11 '16 at 18:39
Is this of any help? http://stackoverflow.com/questions/31306501/python-requests-library-looping-requests-get — Bahrom, Jul 11 '16 at 18:41
@spin_eight if you are typing them manually then perhaps you're doing one per half a second? The loop will do all 6 in 0.1 seconds, likely breaking a rate somewhere. — Tim, Jul 11 '16 at 18:42
@Tim of course I have the script with them typed in manually (this is my fault, I should be more clear about how I do it, I thought that it is evident), so I simply run my script — spin_eight, Jul 11 '16 at 18:45
Looks like you are trying to get the input from a B.S. source like a file (`fIn` is very unclear here). Try explicitly doing `for word in ('set', 'map', 'list', 'vector', 'string'):` for a start. — Mad Physicist, Jul 11 '16 at 18:46
I doubt the issue is the loop itself since I would expect the unlooped commands to run faster, if anything. The problem is most likely with what you are passing to the loop. — Mad Physicist, Jul 11 '16 at 18:47
@Tim. It's being done the same number of times in either case. The looped version is in an `if` block, in a `for` loop, so I would expect it to have quite a bit more overhead. That being said, the overhead will probably be much less than the print itself, as you mentioned. — Mad Physicist, Jul 11 '16 at 18:49
Also, if you don't mind sharing, a sample of `url` that we can use to test. — Mad Physicist, Jul 11 '16 at 18:51
@Bahrom - link you gave my look interesting, but I don't know where to put sessions + close — spin_eight, Jul 11 '16 at 19:03

Padraic Cunningham · Accepted Answer · 2016-07-11T19:05:27.893

3

You have trailing newlines that you need to strip off:

with open('dummyWords.txt') as fIn:
    for word in map(str.strip, fIn) :

It works for the last as you obviously have no newline at the end of the last word in the file. "www.foo.com\n" is not the same as "www.foo.com"

edited Jul 11 '16 at 19:05

answered Jul 11 '16 at 19:03

Padraic Cunningham

176,452
29
245
321

I think standard is you should have a newline at the end of the last line. If he does or not, is different. http://stackoverflow.com/questions/729692/why-should-text-files-end-with-a-newline – Tim Jul 11 '16 at 19:04
@Tim, you are free to edit a file as you please, what you cannot do is add newlines to urls and expect them to work. – Padraic Cunningham Jul 11 '16 at 19:06
My comment refers to the penultimate sentence - he *should* have one there, but he clearly doesn't. – Tim Jul 11 '16 at 19:07
@Padraic Cunningham great!!! I initially thought about it on my way home and this idea somehow vanished! Of course that is it!!! Thank you for you attentive look! Have a nice time of a day!! Best regards! – spin_eight Jul 11 '16 at 19:08
@spin_eight, no worries, you too. – Padraic Cunningham Jul 11 '16 at 19:09

score 0 · Answer 2 · answered Aug 16 '18 at 10:36

0

Encoding POST data resolved issue for me.

cmd = urllib.quote(cmds[i])

For my testcases

No newline required
No sleep time required

answered Aug 16 '18 at 10:36

ajaykools

645
7
12

Python requests module, how to issue multiple requests in the for loop?

2 Answers2

Linked

Related