Return HTTP response and continue processing - python bottle API

Question

I have an analysis service using Python's bottle that is invoked by posting a url to the raw data.

What is the simplest/cleanest/best way to immediately return a 200 response to the callee and continue processing a potentially long-running logic in the application?

Spawn a new thread/process? Use a queue? Async?

from bottle import post, HTTPResponse, request

@post('/postprocess')
def records():
   data_url = request.json.get('data_url', False)
   postback_url = request.json.get('postback_url', False)

   if not data_url or not postback_url:
       return HTTPResponse(status=400, body="Missing data paramater")

   #Immediately return response to callee
   return HTTPResponse(status=200, body="Complete")

   #Continue processing long-running code
   result = get_and_process_data(data_url)
   #POST result to another endpoint

Take a look at Celery or python-rq to run long background tasks. — iurisilvio, Mar 18 '14 at 14:32
I think that you need to use Threads: Read this: [Creating Threads in python][1] [1]: http://stackoverflow.com/questions/2905965/creating-threads-in-python — Rui Martins, Mar 24 '14 at 15:53

score 2 · Accepted Answer · answered Mar 17 '14 at 17:59

2

There is no simplest solution to this. For a production system I would first look into an existing system created for these kinds of situations to determine if that would be a good fit, and if not, only then develop something more suitable to my situation. To that end, I would recommend you take a look at Celery

answered Mar 17 '14 at 17:59

Heikki Toivonen

30,964
11
42
44

Thanks. I ended up using threads but will look at Celery or python-rq for a more robust future solution. – Alyssa Mar 30 '14 at 11:16

score 1 · Answer 2 · answered Mar 22 '14 at 11:38

I would suggest using some Queuing mechanism to queue the data that needs processing. Then you can implement a pool of workers that can work on processing the Q.

You can then take advantage of Q monitoring tools to iron out any performance issues and your worker pool can scale up as needed.

score 1 · Answer 3 · answered Oct 27 '20 at 03:16

I've recently found a very straightforward solution to this.

from threading import Thread    
@app.route('/')  
def send_response():
    uri = request.args
    ua = request.headers['user-agent']
    token = request.args.get("token")
    my_thread = Thread(target=process_response, args=[uri,
                                                      ua,
                                                      token])
    my_thread.start()
    res = Response(status=200,
                   mimetype='application/json')
    return res
        
def process_response(uri, ua, token):
    print(uri)
    print(ua)
    print(token)
    pass

Return HTTP response and continue processing - python bottle API

3 Answers3