3

I have a HTTP-ZeroMQ gateway that accepts POST requests on one end and forwards them on Zmq, then routes the responses back.

The http server is done async with Boost.ASIO and Beast, and runs on a threadpool of 6 threads. Threadpooling is simply done by calling io_service.run() from multiple threads. The zmq part is quite naive: for each http request it creates a new REQ(tcp) socket and synchronously waits for the response.

This clearly has a poor performance as it creates a new zmq/tcp link for each request and the processing thread is blocked until the response arrives (2-3sec). I would like to make the zmq part async as well, but I have some issues. There's a fairly easy way to make zmq sockets ASIO friendly (here and here), I'm more concerned with threading as zmq docs suggest using a the socket from the creating thread only.

I have 3 options in mind:

  1. Have a zmq socket in each threadpool thread with thread_local. In this case I didn't find any way to tell ASIO to run the zmq socket read completion handler on the right thread (which created the socket). If ran in a different thread the handler will see a different thread_local socket, not the one that's available for read.
  2. Similar to above: one socket/thread, but not with thread_local. In this case I don't see any guarantee that the socket is accessed from the right thread, or the socket migration requirement between threads is met
  3. Have a single socket that is protected by a mutex (AFAIK, that does a full memory barrier required for zmq socket migration between threads), but having a single shared resource for all threads doesn't scale well, I don't like it

The server does some data processing as well, so I don't want to make it single threaded.

So, any idea how to elegantly handle zmq sockets async in multi threaded ASIO app?

ps: I did see AZMQ, but I don't want to pull in another 3rd party lib right now.

Community
  • 1
  • 1
Gyorgy Szekely
  • 2,654
  • 3
  • 23
  • 32
  • When you have no external threading requirements with `boost::asio` calling `io_service::run()` from multiple threads is the preferred method. However, if you have external threading requirements (as it appears you do in this case), simply create one `io_service` per thread. – Chad Nov 16 '16 at 15:16
  • This sounds good for constraining the zmq socket completion handler to the socket creator thread. :) But how do I share the http workload in this setup? I have one asio tcp socket on port 8080 whose load is handled by the threadpool in the current design. If I create one io_service/thread then the webserver socket is handled by only one thread... – Gyorgy Szekely Nov 16 '16 at 15:40
  • 1
    I just reviewed the boost asio http server2 example in the asio docs which uses one io_service per thread and round-robin dispatches work to the threads on tcp accept. So this could work. :) – Gyorgy Szekely Nov 16 '16 at 16:24

0 Answers0