7

TLDR: Strands serialise resources shared across completion handlers: how does that prevent the ssl::stream implementation from concurrent access of the SSL context (used internally) for concurrent read/write requests (stream::ssl is not full duplex)? Remember, strands only serialise the completion handler invocation or the original queueing of the read/write requests. [Thanks to sehe for helping me express this better]


I've spent most of a day reading about ASIO, SSL and strands; mostly on stackoverflow (which has some VERY detailed and well expressed explanations, e.g. Why do I need strand per connection when using boost::asio?), and the Boost documentation; but one point remains unclear.

Obviously strands can serialise invocation of callbacks within the same strand, and so also serialise access to resources shared by those strands.

But it seems to me that the problem with boost::asio::ssl::stream isn't in the completion handler callbacks because it's not the callbacks that are operating concurrently on the SSL context, but the ssl::stream implementation that is.

I can't be confident that use of strands in calling async_read_some and async_write_some, or that use of strands for the completion handler, will prevent the io engine from operating on the SSL context at the same time in different threads.

Clearly strand use while calling async_read_some or async_write_some will mean that the read and write can't be queued at the same instant, but I don't see how that prevents the internal implementation from performing the read and write operations at the same time on different threads if the encapsulated tcp::socket becomes ready for read and write at the same time.

Comments at the end of the last answer to this question boost asio - SSL async_read and async_write from one thread claim that concurrent writes to ssl::stream could segfault rather than merely interleave, suggesting that the implementation is not taking the necessary locks to guard against concurrent access.

Unless the actual delayed socket write is bound to the thread/strand that queued it (which I can't see being true, or it would undermine the usefulness of worker threads), how can I be confident that it is possible to queue a read and a write on the same ssl::stream, or what that way could be?

Perhaps the async_write_some processes all of the data with the SSL context immediately, to produce encrypted data, and then becomes a plain socket write, and so then can't conflict with a read completion handler on the same strand, but it doesn't mean that it can't conflict with the internal implementations socket-read-and-decrypt before the completion handler gets queued on the strand. Never mind transparent SSL session re-negotiation that might happen...

I note from: Why do I need strand per connection when using boost::asio? "Composed operations are unique in that intermediate calls to the stream are invoked within the handler's strand, if one is present, instead of the strand in which the composed operation is initiated." but I'm not sure if what I am refering to are "intermediate calls to the stream". Does it mean: "any subsequent processing within that stream implementation"? I suspect not

And finally, for why-oh-why, why doesn't the ssl::stream implementation use a futex or other lock that is cheap when there is no conflict? If the strand rules (implicit or explicit) were followed, then the cost would be almost non-existent, but it would provide safety otherwise. I ask because I've just transitioned the propaganda of Sutter, Stroustrup and the rest, that C++ makes everything better and safer, to ssl::stream where it seems easy to follow certain spells but almost impossible to know if your code is actually safe.

Community
  • 1
  • 1
Sam Liddicott
  • 1,265
  • 12
  • 24

2 Answers2

2

The answer is that the boost ssl::stream implementation uses strands internally for SSL operations.

For example, the async_read_some() function creates an instance of openssl_operation and then calls strand_.post(boost::bind(&openssl_operation::start, op)). [http://www.boost.org/doc/libs/1_57_0/boost/asio/ssl/old/detail/openssl_stream_service.hpp]

It seems reasonable to assume that all necessary internal ssl operations are performed on this internal strand, thus serialising access to the SSL context.

Sam Liddicott
  • 1,265
  • 12
  • 24
  • 1
    Are you sure about that? The file no longer exists. The path includes the word "old". I have searched in the latest Boost.Asio (in Boost 1.67) and I see no strand. – Vinnie Falco May 26 '18 at 22:34
  • @VinnieFalco -- good point. And at over 5 years ago who can be sure of anything...? – Sam Liddicott May 29 '18 at 15:06
  • 3
    @VinnieFalco, actually, there are no internal strands. Moreover, from my experience, scheduling both reads and writes to `ssl::stream` may lead to internal races that either manifest as an SSL communication errors or worse, infinite loop hangups within OpenSSL library. All this considering that operations are issued from a single strand, which obviously of no help. This leads to a sad fact that some async protocols cannot be implemented on top of asio in a multithreaded environment, say HTTP2. – GreenScape Jul 24 '19 at 11:07
  • 2
    Actually the latest ssl::stream documentation says in the thread safety section: The application must also ensure that all asynchronous operations are performed within the same implicit or explicit strand. So I guess calling async_ operations from different threads at the same time on ssl stream are not safe – Gerald Agapov Dec 31 '19 at 13:16
1

Q. but I'm not sure if what I am refering to are "intermediate calls to the stream". Does it mean: "any subsequent processing within that stream implementation"? I suspect not

The docs spell it out:

This operation is implemented in terms of zero or more calls to the stream's async_read_some function, and is known as a composed operation. The program must ensure that the stream performs no other read operations (such as async_read, the stream's async_read_some function, or any other composed operations that perform reads) until this operation completes. doc


And finally, for why-oh-why, why doesn't the ssl::stream implementation use a futex or other lock that is cheap when there is no conflict?

You can't hold a futex across async operations because any thread may execute completion handlers. So, you'd still need the strand here, making the futex redundant.


Comments at the end of the last answer to this question boost asio - SSL async_read and async_write from one thread claim that concurrent writes to ssl::stream could segfault rather than merely interleave, suggesting that the implementation is not taking the necessary locks to guard against concurrent access.

See previous entry. Don't forget about multiple service threads. Data races are Undefined Behaviour


TL;DR

Long story short: async programming is different. It is different for good reasons. You will have to adapt your thinking to it though.

Strands help the implementation by abstracting sequential execution over the async scheduler.

This makes it so that you don't have to know what the scheduling is, how many service threads are running etc.

sehe
  • 374,641
  • 47
  • 450
  • 633
  • 1
    I was expecting that the futex would be held within the ssl::stream implementation while it was operating on the SSL context.This would solve the single-duplex program that ssl::stream has but which tcp::socket does not. I appreciate your general comments on strands and async programming; I don't think they answer the question which is how use of strands when calling async_read_some or async_write_some, or in the expression of the completion handler callback, can solve the duplex problem. – Sam Liddicott Oct 09 '15 at 15:44
  • I would link to the same answer you refer to. I'm honestly not sure what pice of the puzzle is missing (did you spot "_Don't forget about multiple service threads_"?) – sehe Oct 09 '15 at 15:45
  • 2
    You are helping me express the question more clearly, for which I thank you. Strands serialise resources shared across completion handlers: how does that prevent the ssl::stream implementation from concurrent access of the SSL context (used internally) for concurrent read/write requests? Remember, strands only serialise the completion handler invocation or the original queueing of the read/write requests. – Sam Liddicott Oct 09 '15 at 15:55
  • What do you mean with "strands only serialise the completion handler invocation or the original queueing of the read/write requests". That sounds wrong to me. Also, by "serializing" you ought to mean "ensures a happens-before relationship between steps in the sequence" – sehe Oct 09 '15 at 17:12
  • Also strands do not "serialize resources". They serialize (see my definition) operations. Background: `io_service` and `strand` are thread safe anyway (their resources aren't what `strand` synchronizes actions to, because they do this themselves). Strand serializes the operations posted on it (by maintaining a happens-before relation: see http://www.boost.org/doc/libs/1_53_0/doc/html/boost_asio/reference/io_service__strand.html#boost_asio.reference.io_service__strand.order_of_handler_invocation – sehe Oct 09 '15 at 17:25
  • 2
    "What do you mean with "strands only serialise the completion handler invocation or the original queueing of the read/write requests" That sounds wrong to me." I mean that if I strand post async read or async write requests, that they are executed serially and never concurrently, and that the completion handlers are executed serially and not concurrently. Yu are right that strands don't serialise resources, I meant serialise access to resources. However my question is not about either of these, but what occurs internally in ssl::stream. – Sam Liddicott Oct 12 '15 at 08:07