0

Currently I create a library that records backend calls like ones made to boto3 and requests libraries, and then populates a global "data" object based on some data like the status code of responses, etc.

I originally had the data object as global, but then I realized this was a bad idea because when the application is run in parallel, the data object is simultaneously modified (which would possibly corrupt it), however I want to keep this object separate for each invocation of my application.

So I looked into Flask context locals, similar to how it does for its global "request" object. I manage to implement a way using LocalProxy how they did it, so it works fine now with parallel requests to my application - the issue now though, is that whenever the application spawns a new sub-thread it creates an entirely new context and thus I can't retrieve the data object from its parent thread, e.g. for that request session - basically I need to copy and modify the same data object that is local to the main thread for that particular application request.

To clarify, I was able to do this when I previously had data as a true "global" object - multiple sub-threads could properly modify the same object. However, it did not handle the case for simultaneous requests made to application, as I mentioned; so I manage to fix that, but now the sub-threads are not able to modify the same data object any more *sad face*

I looked at some solutions like below, but this did not help me because the decorator approach only works for "local" functions. Since the functions that I need to decorate are "global" functions like requests.request that threads across various application requests will use, I think I need to use another approach where I can temporarily copy same thread context to use in sub-threads (and my understanding is it should not overwrite or decorate the function, as this is a "global" one that will be use by simultaneous requests to application). Would appreciate any help or possible ideas how I can make this work for my use-case. Thanks.

Flask throwing 'working outside of request context' when starting sub thread

rv.kvetch
  • 9,940
  • 3
  • 24
  • 53
  • Have you considered using a database? – noslenkwah Oct 16 '19 at 17:35
  • hmm... it's a good suggestion but I'm not sure how would a database help me, in this case. – rv.kvetch Oct 16 '19 at 17:54
  • I'm unable to get any data from a sub-thread call in my main thread, i think is currently the issue. Basically I want `data` to be a shared object for that request session so I can gather the data and say "these backend calls that ran in sub-threads all had these individual responses" or something similar to that. Currently each sub-thread runs in its own context, so as i understand it they will end up modifying their own `data` object.. no way i have currently to find the data for any sub-threads. – rv.kvetch Oct 16 '19 at 17:58
  • A database solves this exact problem. It is implemented different than a "data object" but it sounds like what you need. – noslenkwah Oct 16 '19 at 19:18
  • sorry, I don't think so.. i just don't see how a database can help out with this. My `data` object can be thought of as very simple object, like a dictionary as so: `{'responses': []}` obviously that's a little simplified, but basically I want the `data` object to be separate per each request session, but only shared between threads of a single request session. Just so that multiple threads can add to the list of "responses", and in the end I can process that data within the scope of the request session. – rv.kvetch Oct 16 '19 at 20:01
  • I'm open to other options or suggestions as well, but basically I would first need some way that I can retrieve the "responses" that each thread will write to (and obviously I need to group them by the request session that they belong to). right now, each thread within a single request context can write to its own data object, but i am basically not able to map it to the request context. I guess I would need to use `get_ident()` to get the main thread's identity, or maybe something similar. – rv.kvetch Oct 16 '19 at 22:26

0 Answers0