0

As the title says, is there a way to run a Jupyter Notebook cell in the background (in Python)?

For example, is there a hypothetical magic command %%run_in_background that allows me to kick off a cell to execute in the background without blocking other cells from executing.

MYK
  • 1,988
  • 7
  • 30
  • Why do you want to do that ? Why don’t you use parallel or multiprocessing – anarchy Jan 06 '22 at 12:36
  • Or 2 different notebooks – anarchy Jan 06 '22 at 12:36
  • @anarchy, sometimes I'm pulling some data from an API. It'd be good if I could continue working in the notebook while that data is being fetched, but it's not worth me spending 5min writing a script that will fetch the data in the background as the query itself will only take 5min, and often depends on the variables I have computed in the notebook. – MYK Jan 06 '22 at 14:25
  • Yeah you’re not meant to do something like that, you need multiple instances. – anarchy Jan 06 '22 at 14:26

1 Answers1

1

Technically you can’t do that, each kernel is like an instance of python, you can only run a script at a time.

You could run the cell, then refresh the browser and run the notebook fresh. The previous command will continue running but there is no way of accessing it in the notebook anymore. You won’t get the result you want, the output from the previous run won’t appear in this new instance.

Your best bet is to use 2 different notebooks, or 2 python scripts or use multiprocessing modules.

anarchy
  • 3,709
  • 2
  • 16
  • 48
  • Is there any way to get the output back? At least what is written to `stdout` after getting it back? – abukaj Dec 22 '22 at 09:04
  • I dont understand, you mean you want to save the output somewhere? – anarchy Dec 22 '22 at 13:41
  • Preferably, but reading its last line would be enough... Or be able to read anything new written to stdout. I detailed the issue in my question: https://stackoverflow.com/questions/74886225/how-can-i-get-stdout-back-in-a-running-cell-of-a-reloaded-jupyter-notebook – abukaj Dec 29 '22 at 13:38
  • can you like explain properly what you are trying to do exactly, maybe I can think of an alternate way to do it. as your question goes, there is no way around it, my answer pretty much sums it up. @abukaj – anarchy Jan 08 '23 at 18:51
  • i think you should be using the multiprocessing, multithreading or async or a mix of these so you can do what you need to do and let python decide which cores or threads you need to use. @abukaj – anarchy Jan 08 '23 at 18:55
  • The proper way (in my case) is to use the `logging` module rather than calling `print()` I guess. But "do not write shitty code" is not an answer here, as what I am interested in is whether (and how) can I rescue anything written to `stdout` after reload of the frontend. Purely technical question. From what you write I understand there is no way. Kernel forwards `stdout` to the non-existing frontend and forgets. Am I right? – abukaj Jan 16 '23 at 15:39