0

I have a program that now will get a REST API. Instead of rewriting a bunch of code I thought I would just run the program, capture the prints and put that into a message response.

On How to capture stdout output from a Python function call? I saw:

  • this solution which is apparently not thread-safe. I tried it with concurrent futures where it worked, but I trust the assessment there that it is not thread safe.
  • I also noticed the implementation here, but while it uses threads to be asynchronous, it never explicitly states that it is itself thread-safe.

This is how I tried to test whether it is thread-safe or not, before I read that it is apparently not thread safe. I never had any issues with this (i.e. output always was ['hello world', 'hello world2'], but maybe this is a characteristic of concurrent futures that is not present when using asynchronous functions from REST API modules (FastAPI in my case).

import concurrent.futures
from io import StringIO 
import sys

def main():
    num_tests = 30
    
    with concurrent.futures.ThreadPoolExecutor() as executor:
        futures = [executor.submit(test) for _ in range(num_tests)]
        
        for future in concurrent.futures.as_completed(futures):
            try:
                result = future.result()
            except Exception as e:
                print(f"An error occurred: {e}")

class Capturing(list):
    def __enter__(self):
        self._stdout = sys.stdout
        sys.stdout = self._stringio = StringIO()
        return self
    def __exit__(self, *args):
        self.extend(self._stringio.getvalue().splitlines())
        del self._stringio    # free up some memory
        sys.stdout = self._stdout
def test():
    with Capturing() as output:
        print('hello world')

    print('displays on screen')

    with Capturing(output) as output:  # note the constructor argument
        print('hello world2')

    print('done')
    print('output:', output)
main()

In the best case I'm looking for an idea how to capture stdout asynchronously. In the worst case an explanation why that isn't possible. In that case I will be looking for other solutions.

jonrsharpe
  • 115,751
  • 26
  • 228
  • 437
Natan
  • 728
  • 1
  • 7
  • 23
  • 2
    `stdout` is a single global variable. How do you think you could replace it in multiple threads and set it to different values? What you can do is call a sub-process and [capture its stdout](https://docs.python.org/3.6/library/subprocess.html#subprocess.CompletedProcess) Of course this has a high per-call overhead of forking the entire process. OTOH it allows true parallelization without worrying about the [GIL](https://stackoverflow.com/questions/1294382/what-is-the-global-interpreter-lock-gil-in-cpython) – Homer512 Aug 08 '23 at 11:33
  • in theory I could think of overwriting the print command globally to something that stores the strings thread-save, but I don't know if that is feasible. – Natan Aug 08 '23 at 11:48
  • 1
    Just how horrible is your code base that you consider monkey-patching `print` rather than replacing a couple `print` and `sys.stdout.write` commands with an extra file object argument? – Homer512 Aug 08 '23 at 11:51
  • It's just that the program itself would remain untouched by the optional REST. The codebase itself is okay, I think. – Natan Aug 08 '23 at 12:23

0 Answers0