1

I'm working on a simple machine learning API being served by Flask (Github repo). My approach so far is:

  • When the app first starts, I have 2 global variables, face_obj and od_obj that load face recognition (dlib) and object detection (YoloV3)models. My goal is to do this only once at the start of the app because model loading is expensive
  • Then, I run the app via app.run(...) and wait for requests. I am running Flask 1.0.2 which I understand defaults to threaded mode, which is what I want to be able to serve requests concurrently
  • I then wait for requests, and depending on the query parameters passed (type=face or type=object) I invoke the .detect() the function of either object and return a JSON feed of what was detected.

The problem I am facing is when my app receives multiple concurrent requests, say 2 object detection requests together, the object detection completely messes up and returns incorrect results. When invoked sequentially, everything is fine.

Here is my main code (api.py):

At the start:

# my two globals 
face_obj = FaceRecog.Face()
od_obj = ObjectDetect.Object()

And then in my detection class, based on API parameters, I invoke either face or object detection:

class Detect(Resource):
    @jwt_required
    def post(self):
        args = parse_args()

        if args['type'] == 'face':
            m = face_obj
            g.log.debug ('Face Recognition requested')

        elif args['type'] in [None, 'object']:
            m = od_obj
            g.log.debug ('Object Recognition requested')

        else:
            abort(400, msg='Invalid Model:{}'.format(args['type']))
        fip,ext = get_file(args)
        fi = fip+ext
        image = cv2.imread(fi)
        detections = m.detect(image)
        return detections

The detection code is standard python detection wrappers using OpenCV or dlib's python wrapper.

I am confused about why detection is getting corrupted. My understanding is that when I launch threads, threads automatically make a copy of objects so there should be no reason for corruption.

I've linked to the complete project above if it helps.

I've read elsewhere on SO (Are global variables thread safe in flask? How do I share data between requests?) that globals are not thread-safe in Flask so should I be converting my face_obj and od_obj instances into session variables? Based on reading that thread, it seems the ask is to share/affect data between threads. In my case, I don't want threads changing data. I just want to pass on the one-time-load model to multiple detection requests.

saeed foroughi
  • 1,662
  • 1
  • 13
  • 25
user1361529
  • 2,667
  • 29
  • 61
  • 1
    Have you tried putting the global variables on top of the code? It is at the bottom currently. – thuyein Jan 19 '20 at 07:25
  • 2
    I think your understanding of the thread is also incorrect. Generally threads will not get a copy of data, they will share the same memory space, only processes will spawn a new copy. [SO answer](https://stackoverflow.com/a/3044626/1509809) – thuyein Jan 19 '20 at 07:28
  • Yeesh, you are so right. Python uses proper threads. It is Perl that does not (or did not). I don't however think the position of globals makes a difference (as long as it is not inside the route handlers, of course) – user1361529 Jan 19 '20 at 12:02
  • I can't say much on threading done by flask, since I mostly done threading through inheriting the `threading.Thread`. But no matter what, due to GIL, the threads should not be able to corrupt anything, as at any point in time, only one thread can be running. Is there any difference, if you run with `threaded=False`? – thuyein Jan 19 '20 at 13:41
  • Thanks. GIL only protects python core structures. Even if a single thread is running at one point of time, given data is shared (as you correctly pointed out) two threads can corrupt the same structure by overwriting half way through their code, for example. I'm not exactly sure _what_ is_ being corrupted - will dive it later this week. Thank you. – user1361529 Jan 20 '20 at 02:17
  • switching to `threaded=False` and `processes=X` solves the issue (but this does mean `X` times the memory as the model is replicated across processes – user1361529 Jan 20 '20 at 05:22

0 Answers0