1

I'm working on a project which needs to connect to a server over a secure socket connection and send some commands, in case one of the commands fail I get an error response on that socket and then the socket closes.

I have tried a single-threaded approach with a non-blocking ssl socket. I connect to the server and then send a few commands, do a non-blocking read for an error status, send some more commands, read again and so on. The server is an Apple Push Notification Service in case it helps.

Everything seems fine until I get an error response followed by the server closing the socket. I can read the error response and see that the socket has been closed but when I free the BIO object the application crashes (unhandled SIGPIPE). In my opinion it crashes because I may be sending data even after the socket has been closed. This is the backtrace after the SIGPIPE signal was issued:

queue = 'com.apple.main-thread', stop reason = signal SIGPIPE
frame #0: 0x94cad46a libsystem_kernel.dylib`__write + 10
frame #1: 0x0007b835 APNTest`conn_write + 69
frame #2: 0x0007957d APNTest`BIO_write + 141
frame #3: 0x00100127 APNTest`ssl3_write_pending + 215
frame #4: 0x001001f9 APNTest`do_ssl3_write + 57
frame #5: 0x00101cab APNTest`ssl3_dispatch_alert + 59
frame #6: 0x000fdb80 APNTest`ssl3_shutdown + 112
frame #7: 0x00126e43 APNTest`ssl_ctrl + 83
frame #8: 0x00005653 APNTest`APNConnection::shutdown(this=0x00300fd0) + 83 at APNConnection.cpp:136
frame #9: 0x000055e5 APNTest`APNConnection::~APNConnection(this=0x00300fd0) + 21 at APNConnection.cpp:41
frame #10: 0x00005687 APNTest`APNConnection::~APNConnection(this=0x00300fd0) + 23 at APNConnection.cpp:40
frame #11: 0x000024d6 APNTest`APNService::run(this=0xbffff7a8) + 1190 at APNService.cpp:73
frame #12: 0x00006a6a APNTest`main + 1946 at main.cpp:36
frame #13: 0x94ce46d9 libdyld.dylib`start + 1

Here is the code I use to create my OpenSSL secure socket

BIO *bio;
SSL_CTX *ctx;
SSL *ssl;

ctx = SSL_CTX_new(SSLv23_client_method());
if (!SSL_CTX_load_verify_locations(ctx, strdup(trustStoreFile.c_str()), NULL)) {...}
if (SSL_CTX_use_certificate_file(ctx, strdup(clientCertFile.c_str()), SSL_FILETYPE_PEM) <= 0) {...}
if (SSL_CTX_use_PrivateKey_file(ctx, strdup(clientPKFile.c_str()), SSL_FILETYPE_PEM) <= 0) {...}

bio = BIO_new_ssl_connect(ctx);
BIO_get_ssl(bio, & ssl);
SSL_set_mode(ssl, SSL_MODE_AUTO_RETRY);
BIO_set_nbio(bio, 1);
BIO_set_conn_hostname(bio, serverURL.c_str());
int connErr = 0;
do {
    connErr = BIO_do_connect(bio);
} while (BIO_should_retry(bio));

Reading and writing data is performed using BIO_write(bio, data, size) and BIO_read(bio, data, datasize)

Cleanup code and the one that crashes is 

SSL_CTX_free(ctx);
BIO_reset(bio);
BIO_free_all(bio);

BIO_reset is the one that crashes right now, but if I remove it BIO_free_all will crash. What I can do to avoid the crash is send one command at a time and after each command is sent I should wait 1 second or so to see if the server sent an error for that command or not, send one more, wait again and so on. Or another thing that helped me hide the crash is installing a dummy signal handler for SIGPIPE but these are just workarounds.

I'm guessing that it may be crashing because of the data queued for sending after the socket was closed. What makes me believe this is that I see that ssl3_shutdown calls some ssl3_write_pending and also because the cleanup code will not crash if I am extra careful to not do any more BIO_write(s) after I get a non-zero SSL_shutdown().

Am I doing something wrong with the cleanup code or is there a way to instruct OpenSSL to discard any data when freeing the BIO and SSL objects?

Thank you for your help!

mihair
  • 11
  • 2
  • Usually you get a SIGPIPE if the socket connection or PIPE is broken i.e. the other side is no longer present in which case, in most cases the application should use the `SIG_IGN` for handling the signal for example `signal(SIGPIPE, SIG_IGN);` . Please look at the following answer for detail. http://stackoverflow.com/questions/108183/how-to-prevent-sigpipes-or-handle-them-properly. `SIG_IGN` will ignore the signal. – cmidi Mar 11 '15 at 14:51
  • Changing the signal handler for the broken pipe signal only hides the problem that I'm dealing with. Right now the code is in a dev testing stage and I can get away with global changes like ignoring a signal but later on it will be integrated with a bigger product which may want to add signal handlers of it's own. – mihair Mar 11 '15 at 15:07
  • what you cannot control is the other end of the connection from breaking the connection which would lead you to have a `SIGPIPE` signal which I think can be ignored. what cannot be ignored is the error the socket write returns which is `errno 32 ` on most linux systems `broken pipe` in which case you need to handle the error correctly and stop writing to it. – cmidi Mar 11 '15 at 15:54
  • I've just checked and BIO_write() always returns the number of bytes written(which is around 100 bytes per command) and never returns an error. I must remind you that I'm using secure sockets and thus data is not immediately pushed to the socket but it's buffered and encoded. Maybe this is why BIO_write always succeeds...maybe? – mihair Mar 11 '15 at 16:10
  • are you using openssl ? – cmidi Mar 11 '15 at 16:18
  • maybe `ssl_write` or `bio_write` does not have any error. The error arises when you do a `SSL_shutdown` which in turn tries to send a `close_notify` message if I remember the proto.Which in turn is causing the broken pipe error probably because the other side is already closed "maybe because of wrongly written SSL code far end". Handling SSL_shutdown could be a tricky thing. you can look at the following links for more understanding. provided you are using openssl http://stackoverflow.com/questions/28056056/handling-ssl-shutdown-correctly https://www.openssl.org/docs/ssl/SSL_shutdown.html – cmidi Mar 11 '15 at 16:30
  • Yes, I am using OpenSSL, v1.0.2 to be exact. I'm expecting from a mature library like this to not do this kind of rookie mistakes like writing to a socket it already knows is closed. I'm thinking that I may be using the interface incorrectly which causes the crash – mihair Mar 11 '15 at 16:39
  • when I said wrongly written far end I did not mean openssl I meant the client/server written using some ssl library one such case is if the other side just ends the connection by calling `close(fd)` without calling `SSL_shutdown` or socket `shutdown(fd,x)` or the application on the far end just crashes which then would cause the connection to break and cause broken pipe error. openssl would have no idea about what happened on the other end but to just return the application with a `SSL_ERROR_SSL` or a `SSL_ERROR_SYSCALL` so the application has to check the error stack or return errno value – cmidi Mar 11 '15 at 20:00
  • the tcp state machine would not know that the far side has gone until it sends a packet out provided it has not received a `FIN/FIN-ACK` and tcp timers have kicked in so there is no way for openssl to know that – cmidi Mar 11 '15 at 20:08

0 Answers0