How to diagnose why Ctrl + C not stopping pserve

Question

I'm experimenting porting a project from Cherrypy to Pyramid web framework. I have a small part converted and notice that Ctrl+C will not stop the Pyramid application. A cookiecutter version will stop with Ctrl+C. I end up needing to kill the process every time.

I am serving using the pserve command that uses the waitress WSGI server in both cases...

pserve development.ini

I should also note: I am running Debian Stretch in a VirtualBox VM.

Is there a way to know why the behavior has changed or how to restore Ctrl+C shutdown? How could I know if something is now blocking this from happening?

-- Additional Information asked for in comments --

Using grep Sig /proc/process_id/status yields the following:

SigQ:   0/15735
SigPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000001001000
SigCgt: 0000000180004002 hex/binary 110000000000000000100000000000010

Using GDB and getting a py-bt

    (gdb) py-bt
Traceback (most recent call first):
  <built-in method select of module object at remote 0x7f914f837e58>
  File "/usr/lib/python3.5/asyncore.py", line 144, in poll
    r, w, e = select.select(r, w, e, timeout)
  File "/usr/lib/python3.5/asyncore.py", line 203, in loop
    poll_fun(timeout, map)
  File "/home/clutton/programs/python/webapps_pyramid/env/lib/python3.5/site-packages/waitress/server.py", line 131, in run
    use_poll=self.adj.asyncore_use_poll,
  File "/home/clutton/programs/python/webapps_pyramid/env/lib/python3.5/site-packages/waitress/__init__.py", line 17, in serve
    server.run()
  File "/home/clutton/programs/python/webapps_pyramid/env/lib/python3.5/site-packages/waitress/__init__.py", line 20, in serve_paste
    serve(app, **kw)
  File "/home/clutton/programs/python/webapps_pyramid/env/lib/python3.5/site-packages/paste/deploy/util.py", line 55, in fix_call
    val = callable(*args, **kw)
  File "/home/clutton/programs/python/webapps_pyramid/env/lib/python3.5/site-packages/paste/deploy/loadwsgi.py", line 189, in server_wrapper
    **context.local_conf)
  File "/home/clutton/programs/python/webapps_pyramid/env/lib/python3.5/site-packages/pyramid/scripts/pserve.py", line 239, in run
    server(app)
  File "/home/clutton/programs/python/webapps_pyramid/env/lib/python3.5/site-packages/pyramid/scripts/pserve.py", line 32, in main
    return command.run()
  File "/home/clutton/programs/python/webapps_pyramid/env/bin/pserve", line 11, in <module>
    sys.exit(main())

I'm not familiar with the projects you're working with, but the two ways I know to catch Ctrl+C [are outlined here](https://stackoverflow.com/questions/1112343/how-do-i-capture-sigint-in-python). — jedwards, Oct 16 '18 at 00:14
Thanks for that but I'm really more so interested in how to know what is blocking something that seems to work by default, how to trace it down (GDB perhaps?) or how to know why it quit working... — clutton, Oct 16 '18 at 00:18
Ctrl+C signal delivery is more complex than your average signal, involving terminals and process groups. When you next have a chance to kill the process, can you try `kill -2 processid` and let us know if that works? — Mark Plotnick, Oct 16 '18 at 00:40
Neither of kill -s sigint(2)/ kill -s sigterm(15) work.... only kill -s sigkill (9) will end the process. It seems like something blocks but I can't seem to figure out how to figure out what it is that is blocking... — clutton, Oct 16 '18 at 00:47
How do you start each of the two applications? What is different between what you ported and the cookiecutter version? — Steve Piercy, Oct 16 '18 at 01:24
What is the full uwsgi command you are using to run the server? — Hassan Voyeau, Oct 16 '18 at 15:33
@HassanVoyeau - No signals seem to work other than SIGKILL. It is being hosted by Waitress not uWSGI. Really looking for debug methods to find what could be blocking as I think that is the issue. — clutton, Oct 16 '18 at 15:56
I am guessing something in your stack is being run in the background hence why Ctrl + C is not working. What does your .ini file look like? — Hassan Voyeau, Oct 16 '18 at 16:42
Can you run `grep Sig /proc/processid/status` and look at the last hex digit in each line and tell us whether the 2nd-lowest bit is on in any of them? That will show if SIGINT is being caught or ignored or something else. — Mark Plotnick, Oct 17 '18 at 19:29
@MarkPlotnick - I have added the information you requested to the question. — clutton, Oct 17 '18 at 19:56
OK, it shows that that process has a signal handler for SIGINT. Is that process something that you have source code for? If you're not sure, attach to it with `gdb` and get a backtrace with `bt` to see if anything looks familiar. — Mark Plotnick, Oct 17 '18 at 20:31
@MarkPlotnick - Since it is python it shows all the python c function calls... I can do a py-bt and have attached it to the question. — clutton, Oct 17 '18 at 22:19

score 2 · Answer 1 · answered Oct 18 '18 at 16:46

In order to diagnose where I was running into issues I took the following steps guided by many of the comments made on the question.

I inspected the process to ensure that signals were indeed being caught.

grep Sig /proc/process_id/status

Which yields the following information:

SigQ:   0/15735  
SigPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000001001000
SigCgt: 0000000180004002 hex/binary 110000000000000000100000000000010

The SigCgt indicates signals that are indeed being listened to, in the above the hex value converted to binary shows that (from right to left) signal 2 and 15 were indeed bound.

At this point we need to diagnose why there would be a handler yet it appeared to not be working. So the remaining question was what was the handler. To find that out I used the Python signal module and added some code where I could see it in a debugger...

import signal
s = signal.getsignal(2)

Once I did that I found that the handler referenced a function from a standalone script that is part of the project. I was overwriting the default signal handlers in order to do cleanup before terminating the process but... I was importing it as well in part of this project that had it's own process. Since the project is normally developed on Windows, I was probably dealing with different signals when using Ctrl-C previously so this bug has existed for a long time and doing some Linux development work for the project brought it to light.

How to diagnose why Ctrl + C not stopping pserve

1 Answers1