Multiple accidental POST requests in Python

Question

I have this program that sends a GET request using requests library and then sends a POST request to another server.

import requests 

# inside the loop : 
headers = {'charset':'utf-8','Content-Type':'application/json'}
url = "http://someurl/_get_v2.php"
data = jenal

try :
    resp = requests.post(url,json=data,headers=headers, timeout=(connect_timeout,read_timeout))

    print "Post request sent" 
# If a connection error occurs, start from the beginning of the loop
except requests.exceptions.ConnectionError as e: 
    continue 
# If a read_timeout error occurs, start from the beginning of the loop
except requests.exceptions.ReadTimeout as e:  
    continue
time.sleep(10)

This is the POST part from the main loop. I used try and except in case if the program runs into a delay, I don't want it to stop for any error, but refreshes and continues from the beginning of the loop and so on for ever. Logically, the program should be fine, but when there is a multiple number of delays suddenly the program sends multiple POST requests at the same time.

E.g: the log should be like this: 10 sent. 20 sent. 30 sent. 45 sent (5 seconds delay). 55 sent

but what happens is this : 10 sent 20 sent 30 sent delay..
45 sent 45 sent 45 sent (a few copies of the data are sent at the same time which corrupts my database) 55 sent

How am I supposed to prevent the extra copies?

This is the first part of the program. In case there's anything in the post part, this part repeats and I can see the print each time:

connect_timeout = 3.05
read_timeout = 2

while True:
loop_time_start = time.time()
# GET

url_source = 'http://something/api_json.asp?cmd=list_metering&auth=YWRtaW46YWRtaW4='
try: 
    url = requests.get(url_source)
    print "GET request sent" 
except requests.exceptions.ConnectionError as e: continue

What if you add a `boolean` to it and when POSTing, set it to `True`.. `posting = True`, and when you catch an error or finish POSTing, set it to `False`.. `posting = False`. So then you can say `if not posting:`. — Dobz, Jul 03 '15 at 07:37
put it outside the `try:`, `if not posting: try: posting = True`. Then after your `print` and inside the `except`s put in the `posting = False`... On second thought, it may be the continues, it doesn't give time to wait for next one. — Dobz, Jul 03 '15 at 07:44

Dobz · Answer 1 · 2015-07-03T08:37:28.133

1

What if you add a timer to the code.

import requests 
from datetime import datetime, time

posting = False

# inside the loop :

if not posting:
    posting = True

    headers = {'charset':'utf-8','Content-Type':'application/json'}
    url = "http://someurl/_get_v2.php"
    data = jenal

    start = datetime.now()
    try :
        resp = requests.post(url,json=data,headers=headers,timeout=(connect_timeout,read_timeout))

        print "Post request sent" 

        posting = False

    except requests.exceptions.ConnectionError as e:
        # If you continue here, then there will be no sleep.
        # If your POST fails straight away, it tries again straight away. 

        # You should print/log here so you know which exception is being caught.

        # Add Timer
        end = datetime.combine(start.date(), time(0))

        ShouldISleep(start, end)

        posting = False

        continue

    except requests.exceptions.ReadTimeout as e:
        # If you continue here, then there will be no sleep.
        # If your POST fails straight away, it tries again straight away.

        # You should print/log here so you know which exception is being caught.

        # Add Timer
        end = datetime.combine(start.date(), time(0))

        ShouldISleep(start, end)

        posting = False

        continue

    time.sleep(10)

Create a function

def ShouldISleep(t1, t2):
    passed = (t1 - t2).seconds
    if passed < 10:
        time.sleep(10 - passed)

edited Jul 03 '15 at 08:37

answered Jul 03 '15 at 07:47

Dobz

1,213
1
14
36

That's true sir, I want it to start from the beginning of the loop in case if there's an error, is it what's happening? Until now I don't understand this bug :? I want to sleep only after sending, but the multiple posts are the problem. – Ahmed Al-haddad Jul 03 '15 at 07:55
What about doing a small sleep if it fails? Just to be sure there is a delay. It seems like it's failing straight away and trying again straight away, so it's constantly bombarding POSTs. You could even sleep for a second and check does this help. – Dobz Jul 03 '15 at 07:57
Actually the delay is already there sir but I didn't include that part of the program, but let me include it in the question. – Ahmed Al-haddad Jul 03 '15 at 08:02
I just checked and errors can happen more than once and they can combine both read timeout and connect timeout – Ahmed Al-haddad Jul 03 '15 at 08:18
1

Is this running asynchronously? It seems like there are multiple version running at once, which may be fixable with the boolean. – Dobz Jul 03 '15 at 08:32
1

I've added the booleans, so that if there is currently a POST being made, another one cannot run. – Dobz Jul 03 '15 at 08:38
Could you please tell me which part is it that shows the multiple versions? I still can figure out the exact source but I tried changing some of the variables by deleting `data = jenal` and unifying the timeout delay to a constant of 3.05 and it showed better results when testing it. Do you think that was the problem?! – Ahmed Al-haddad Jul 06 '15 at 01:01
I was testing the program just now. When lag timeout delays occurred, I was also writing the data to a text file. The text file had only one reading while the server had multiple of readings. Does this mean that the problem is in the library `requests` itself?! – Ahmed Al-haddad Jul 07 '15 at 04:44
1

I highly doubt there is a problem in the requests library. Heres a link that may help you further your knowledge on the request library, and help debug it. :) http://www.pythonforbeginners.com/python-on-the-web/using-requests-in-python/ – Dobz Jul 08 '15 at 11:44
Thank you so much sir for the link. I have found out that this happens mostly when there's a read timeout error. In most cases it will send the POST request but there will be no response from the server, so I will send it again, while the first was sent already. This results in two entries when the server finally responds, or more depending on how many times I resent the request. For this kind of case, do you have any suggestion that I can implement to prevent this kind of behavior?! – Ahmed Al-haddad Jul 09 '15 at 01:33
1

You should run these Asynchronously and let them timeout themselves. Here is a link that shows how to run a `GET` asyncronously using the `request` library. You should be able to follow the same instructions, but change them to a `POST`. :) http://stackoverflow.com/questions/9110593/asynchronous-requests-with-python-requests – Dobz Jul 09 '15 at 07:58

score 0 · Accepted Answer · edited May 23 '17 at 12:29

0

I would recommend looking at the approved answer on this question.

edited May 23 '17 at 12:29

Community

1
1

answered Jul 08 '15 at 08:42

Ahmed Al-haddad

805
2
16
41

Multiple accidental POST requests in Python

2 Answers2

Linked