0

I am running a large job. It has been going for more than a week. I had the foresight to pickle off some of its state every once in a while, but I would like access to a particular list that I neglected to periodically pickle.

If I end execution of the program, I will lose the state of this list. If I rewrite my script to also pickle the list, then that does me no good, since I would have to restart from scratch and lose access to my existing variables.

Ctrl+z can pause a program, and fg will resume the job. Is there a way to examine the paused state or gain access to the memory space of the process, running or paused?

Pavel Komarov
  • 1,153
  • 12
  • 26
  • For these long running task I would recommend using task queue, like Celery: http://www.celeryproject.org/. The task queue can be monitored and is even persistent (=survives crash/reboot). – Andrej Kesely Jul 12 '18 at 16:15
  • I can attach gdb with `gdb -p `, which pauses the execution of my script and gives me some low-level access to memory. But I am not sure what to do from there. – Pavel Komarov Jul 12 '18 at 17:11
  • I would suggest then some tutorial about gdb or pdb (Python debugger). Try https://www.youtube.com/watch?v=bZZTeKPRSLQ&t=350s – Andrej Kesely Jul 12 '18 at 17:18
  • pdb seems great, but how can I attach it to a running process? I've tried using the `py-locals` command to no avail, which might mean my python was installed without debug flags. – Pavel Komarov Jul 12 '18 at 17:29
  • Can I make use of any of the code here? https://github.com/python/cpython/blob/master/Tools/gdb/libpython.py – Pavel Komarov Jul 12 '18 at 17:35
  • More info how to attach to process: https://stackoverflow.com/questions/25308847/attaching-a-process-with-pdb – Andrej Kesely Jul 12 '18 at 17:35
  • Yeah, I think I need the debug symbols and a gdb that is compiled with the -with-python flag. https://sumitkgaur.wordpress.com/2014/05/13/python-debugging/ https://www.youtube.com/watch?v=rB9rPdMRxIA Unfortunately, I don't have either of these things, and my environment isn't managed by me so I can't just install them, and even if I could the existing process is running in a version of python without debug symbols. I have to literally use the low-level gdb commands and reconstruct the python somehow. Where exactly in memory should I look? How can I make a python object out of those bits? – Pavel Komarov Jul 12 '18 at 18:06

0 Answers0