1
import requests
while True: 
    try:
        posting = requests.post(url,json = data,headers,timeout = 3.05)
    except requests.exceptions.ConnectionError as e: 
        continue
    # If a read_timeout error occurs, start from the beginning of the loop
    except requests.exceptions.ReadTimeout as e:  
        continue

a link to more code : Multiple accidental POST requests in Python This code is using requests library to perform POST requests indefinitely. I noticed that when try fails multiple of times and the while loop starts all over multiple of times, that when I can finally send the post request, I find out multiple of entries from the server side at the same second. I was writing to a txt file at the same time and it showed one entry only. Each entry is 5 readings. Is this an issue with the library itself? Is there a way to fix this?! No matter what kind of conditions that I put it still doesn't work :/ ! You can notice the reading at 12:11:13 has 6 parameters per second while at 12:14:30 (after the delay, it should be every 10 seconds) it is a few entries at the same second!!! 3 entries that make up 18 readings in one second, instead of 6 only!

Community
  • 1
  • 1
Ahmed Al-haddad
  • 805
  • 2
  • 16
  • 41
  • 2
    What is the expected functionality here? – Sait Jul 07 '15 at 05:12
  • The code is supposed to send a GET request and then POST it and repeat. Sometimes due to network issues, the POST requests fails a few times so I repeat for 2 times, if posted or else GET and try to POST again. Sometimes when trying to POST but fails, after a few tries, it finally POSTs but bursts a set of entries instead of one entry only.. Like shown in the picture. The picture shows the server side. – Ahmed Al-haddad Jul 07 '15 at 05:16
  • 1
    I think we need more code. Also, please fix your indentations - they are important in Python! Finally, can you look at the arguments in your `.post` method again? They are pretty messed up. – justinpawela Jul 07 '15 at 05:20
  • @user2194039 I will work on providing the code in a bit, but what's wrong with my `.post`method? Could you please show me where is the mistake with the indentation? The code works except that it gives me this irregular result. – Ahmed Al-haddad Jul 07 '15 at 05:29
  • 1
    I assume everything after `while True:` should be indented another level. And in Python, you must have all your non-keyword arguments (like `url` and `headers`) before your keyword arguments (like `json = data` and `timeout = 3.05`). But I actually assume you meant `.post(url, data=json, headers=headers, timeout=3.05)`. In any case, your code definitely will not run as written in the question. – justinpawela Jul 07 '15 at 05:36
  • @user2194039 I just tried `data = json` and it didn't work out. I suppose it is like this to encode it before sending it. – Ahmed Al-haddad Jul 07 '15 at 06:01
  • 1
    No, stick with `json=data`, I just flipped it around. – justinpawela Jul 07 '15 at 06:02
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/82561/discussion-between-ahmed-al-haddad-and-user2194039). – Ahmed Al-haddad Jul 07 '15 at 06:05

1 Answers1

2

It looks like the server receives your requests and acts upon them but fails to respond in time (3s is a pretty low timeout, a load spike/paging operation can easily make the server miss it unless it employs special measures). I'd suggest to

  • process requests asynchronously (e.g. spawn threads; Asynchronous Requests with Python requests discusses ways to do this with requests) and do not use timeouts (TCP has its own timeouts, let it fail instead).
  • reuse the connection(s) (TCP has quite a bit of overhead for connection establishing/breaking) or use UDP instead.
  • include some "hints" (IDs, timestamps etc.) to prevent the server from adding duplicate records. (I'd call this one a workaround as the real problem is you're not making sure if your request was processed.)

From the server side, you may want to:

  • Respond ASAP and act upon the info later. Do not let pending action prevent answering further requests.
Community
  • 1
  • 1
ivan_pozdeev
  • 33,874
  • 19
  • 107
  • 152
  • 1
    Thanks a lot for your elaborate answer. I used your advice yesterday about removing the timeout and surprisingly I still got the same problem, but on a smaller scale (only two entries at one time). However, seems like this problem only occurs when there's a readtimeout exception from my observations. I am still going to employ and test asynchronous requests soon, but thought of letting you know and see if you have got any input regarding this update. – Ahmed Al-haddad Jul 08 '15 at 08:29
  • 1
    1) By eliminating `timeout`, you only stopped interfering with TCP's delivery efforts (thus, naturally, getting better delivery). Nothing would insure you from server not responding in _any_ time frame, it's only you to decide how long to try before ultimately giving up. 1.1) If you have access to server, you may diagnose the problem from its side to find out what takes it so long. 2) The "two entries at one time" (rather than one) can be caused by you not using async I/O: if I/O took long enough, your program may have missed the time for the next send and jumped straight to the following one. – ivan_pozdeev Jul 08 '15 at 12:20
  • I have studied asynchronous requests and now I what you mean. I have found out that the problem happens mostly when there's a read timeout error. In most cases it will send the POST request but there will be no response from the server, so I will send it again, while the first was sent already. This results in two entries when the server finally responds, or more depending on how many times I resent the request during the same problematic period of timeout. By using keep-alive to reuse, does that means that I need to create a session for both GET and POST to maintain the same connection? – Ahmed Al-haddad Jul 09 '15 at 04:52
  • Basically atm I am trying to minimize the multiple entries as seemingly I have no power over a network breakdown etc regardless of the lost data. But since a read timeout causes this, maybe there's something that I can do? Isn't using keep-alive/Session() will still allow multiple entries within the same connection?! Or there's a way to check if the connection is established without a read timeout and then I can send the data?! – Ahmed Al-haddad Jul 09 '15 at 05:50