ASIO signal_set not reliable with multiple IO threads, depending on code order?

Question

Edit: I can no longer reproduce this problem. Without changing anything, the signal_set works reliably regardless of the ordering of blocks now.

I am using (standalone) ASIO in a program, and in order to shutdown gracefully on Ctrl+C, I use a signal_set. Everything works well when only my main thread calls io_context.run().

Then, I added an option to use several threads for IO. It looks something like this:

// begin block 1
asio::signal_set signals(io_context, SIGINT, SIGTERM);
signals.async_wait([&server, &signals] (const asio::error_code& ec, int signal) {
    std::cerr << "Received signal " << signal << ", exiting" << std::endl;
    server.shutdown();
    signals.clear();
});
// end block 1

// begin block 2
std::vector<std::thread> io_threads;
if (num_io_threads > 1) {
    for (int i = 1; i < num_io_threads; ++i) {
        io_threads.emplace_back([&io_context] () {io_context.run();});
    }
}
// end block 2

io_context.run();

for (auto& thread: io_threads) {
    thread.join();
}

However, when I ran with num_io_threads > 1 and press Ctrl+C, the programs stopped abruptly instead of shutting down gracefully. I thought it may be because the extra threads "stole" the signal, since I have not masked any signal out in those threads.

Then I had a hunch, and reordered the code, moving block 1 below block 2, and sure enough, graceful shutdown works reliably again.

Is this behavior something I can rely on? Specifically, is it because that I created the signal_set and called its async_wait method after creating all threads, that the signal callback is triggered reliably, or is it because of something else? If it's something else, what's the proper solution of having the signal callback triggered reliably?

I tried to find relevant documentation, but couldn't find any. The docs only say that programs must ensure that any signals registered using signal_set objects are unblocked in at least one thread.

Everything is on CentOS 7 with g++ 4.8.5.

score 2 · Accepted Answer · answered Jun 22 '20 at 13:13

Yes you can rely on it.

I'm personally mildly surprised that you saw the effect you report with the blocks in order (#1,#2).

I can't reproduce it either:

Live On Coliru

#include <boost/asio.hpp>
#include <iostream>

namespace boost::asio {
    using boost::system::error_code; // huh - maybe this is a non-boost Asio thing
}
namespace asio = boost::asio;

template <typename Executor> struct Server {
    Server(Executor ex)
            : s(make_strand(ex)),
              timer(s, std::chrono::high_resolution_clock::time_point::max())
    {
        timer.async_wait([](asio::error_code ec) {
            std::cout << "Server shutdown (" << ec.message() << ")" << std::endl;
        });
    }
    void shutdown() {
        post(s, [this] { timer.cancel(); });
    };

  private:
    asio::strand<Executor> s;
    asio::high_resolution_timer timer;
};

int main(int argc, char**) {
    std::vector<std::thread> io_threads;
    boost::asio::io_context io_context;
    const int num_io_threads = 30;

    Server server(io_context.get_executor());

    auto start_threads = [&io_threads, &io_context] { //"block #2"
        // "block 2"
        if (auto n = num_io_threads - (io_threads.size() + 1); n > 0) {
            std::cerr << "Starting " << n << " threads...\n";
            while (n--)
                io_threads.emplace_back([&io_context] { io_context.run(); });
        }
    };

    if (argc > 1)
        start_threads();

    std::cerr << "Starting signal_set...\n";
    // begin block 1
    asio::signal_set signals(io_context, SIGINT, SIGTERM);
    signals.async_wait(
        [&server, &signals](const asio::error_code& ec, int signal) {
            std::cerr << "Received signal " << ::strsignal(signal) << ", " << ec.message() << std::endl;
            if (!ec)
            {
                std::cerr << "Exiting" << std::endl;
                server.shutdown();
                signals.clear();
            }
        });
    // end block 1

    start_threads();

    io_context.run();

    for (auto& thread : io_threads) {
        thread.join();
    }
}

Which runs both orderings with equal "success":

./a.out        & sleep 1; kill -INT $!
./a.out order2 & sleep 1; kill -INT $!
Starting signal_set...
Starting 29 threads...
Received signal Interrupt, Success
Exiting
Server shutdown (Operation canceled)
bash: fork: retry: Resource temporarily unavailable
Starting 29 threads...
Starting signal_set...
bash: fork: retry: Resource temporarily unavailable
Received signal Interrupt, Success
Exiting
Server shutdown (Operation canceled)

Some thoughts:

signal_set isn't thread safe, so be sure you don't access it concurrently.
Same idea goes for server.shutdown(). In my repro I made the shutdown post on a strand to avoid races.
I added a check on ec in the signal handler
You really ought to handle exceptions in the io threads: https://stackoverflow.com/a/44500924/85371
Even simpler, consider using asio::tread_pool (Coliru)

Summary

If you can reproduce with the above code, I suspect there is a (platform dependent?) bug in the signal set service implementation, worth reporting/asking the Asio developers.

So you're saying that the order should not have mattered, and the signal_set should work reliably in either ordering? — twisteroid ambassador, Jun 23 '20 at 07:21
I didn't change anything, but now the signal_set works regardless of the order of the two blocks. Weird. — twisteroid ambassador, Jun 23 '20 at 08:56

ASIO signal_set not reliable with multiple IO threads, depending on code order?

1 Answers1

Summary