5

I'm writing the code to attend a service. I receive POST requests to a particular service and start processing. The whole process is rather simple; I loop trough items in the request and add each item to the database. The problem arise when I have to process a lot of items and the loop takes like three minutes to finish, then when I try to respond:

status = '200 OK'
headers = [('Content-type', 'application/json'),('Access-Control-Allow-Origin','*')]
start_response(status, headers)

return json.dumps(response)

I get this error:

Exception happened during processing of request from ('XXX.XXX.XXX.XXX', 49172)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/SocketServer.py", line 284, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/local/lib/python2.7/SocketServer.py", line 310, in process_request
    self.finish_request(request, client_address)
  File "/usr/local/lib/python2.7/SocketServer.py", line 323, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/local/lib/python2.7/SocketServer.py", line 640, in __init__
    self.finish()
  File "/usr/local/lib/python2.7/SocketServer.py", line 693, in finish
    self.wfile.flush()
  File "/usr/local/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe

I don't know if this helps, but the POST request is a forwarded POST made from a browser to a different domain (that's why the a Access-Control-Allow-Origin) and all the accesses to the database are made using a single object that interacts with the database using SQLAlchemy (can be seen similar to a Java EE DAO pattern).

How do I avoid this error?

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Sergio Ayestarán
  • 5,590
  • 4
  • 38
  • 62
  • possible duplicate of [Django development server broken pipe error](http://stackoverflow.com/questions/10253288/django-development-server-broken-pipe-error) – miku Jan 04 '13 at 21:02
  • You can make this asynchronous. As soon as all the data is here, you send back a token saying the data is received and at the client side you can display a "Loading..." message. Make the client periodically check for updates if the data has finished uploading, and then once finished, you display the full ack. – Sudipta Chatterjee Jan 04 '13 at 21:11
  • Usually when I get a broken pipe error, its something stupid, like the client timed out while I was debugging the server code. If the process takes more than three minutes, the process should be done asynchronously and you should respond confirming that the process started successfully instead of waiting until the process is done running to respond. – Chris Dutrow Jan 05 '13 at 01:33

2 Answers2

5

You maybe are violating the idea behind REST.

If the proccessing could take some time, the service may want to answer it with an 202 Accepted Response! For a full overview of http response codes follow this link: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

202 Accepted

The request has been accepted for processing, but the processing has not been completed. The request might or might not eventually be acted upon, as it might be disallowed when processing actually takes place. There is no facility for re-sending a status code from an asynchronous operation such as this.

The 202 response is intentionally non-committal. Its purpose is to allow a server to accept a request for some other process (perhaps a batch-oriented process that is only run once per day) without requiring that the user agent's connection to the server persist until the process is completed. The entity returned with this response SHOULD include an indication of the request's current status and either a pointer to a status monitor or some estimate of when the user can expect the request to be fulfilled.

Community
  • 1
  • 1
sschrass
  • 7,014
  • 6
  • 43
  • 62
  • You may be right, a 202 sounds like a good idea, still how can I respond with a 202 and start processing the request at the same time? – Sergio Ayestarán Jan 04 '13 at 23:10
  • typically you launch a thread. Upon its completition it should create a resource, that can be accessed by get-requests. – sschrass Jan 04 '13 at 23:14
  • In this case, launching an asynchronous process is what you want, if the request takes a long time to answer. The 202 code seems like the best solution. So, you launch the process, then return 202 and continue processing in the background. – Juan Carlos Coto Jan 06 '13 at 02:30
2

I could be wrong, but it just looks to me like the socket is timing out. You shouldn't leave the client hanging for a response for more than three minutes.

Instead, you should validate the data and send a message stating that it was received. If necessary, you can use something like AJAX to tell the client that the data was entered after it was received.

Fredrick Brennan
  • 7,079
  • 2
  • 30
  • 61