I'm debugging an issue which only seems to happen on Solaris (we test on Windows, Linux, OSX, Solaris, HP-UX, and AIX).
Our code throws an exception (which is totally expected), and this results in the 'terminate called after throwing an instance of ...' message being printed, and the process being terminated.
(dbx) where -h
current thread: t@1
dbx: forward reference `Connection' not defined, see `help forwardref'
[1] __lwp_kill(0x0, 0x6, 0x0, 0x6, 0xffbffeff, 0x0), at 0xff2ce364
[2] raise(0x6, 0x0, 0x0, 0xff2ad954, 0xffffffff, 0x6), at 0xff267958
[3] abort(0x132cb0, 0x1, 0xff122f38, 0xedbe0, 0xff333558, 0x0), at 0xff242910
=>[4] __gnu_cxx::__verbose_terminate_handler(), line 95 in "vterminate.cc"
[5] __cxxabiv1::__terminate(handler = 0xff125098 = &__gnu_cxx::__verbose_terminate_handler()), line 47 in "eh_terminate.cc"
[6] std::terminate(), line 57 in "eh_terminate.cc"
[7] __cxa_throw(obj = 0x132cf8, tinfo = 0xfef2bf2c, dest = 0xfe204240 = &Simba::Support::ErrorException::~ErrorException()), line 87 in "eh_throw.cc"
[8] Simba::ODBC::ConnectionAttributes::SetAttribute(this = 0x12a060, in_attribute = <value unavailable>, in_value = 0x132c78, in_length = -3), line 299 in "ConnectionAttributes.cpp"
Simba::Support::ErrorException
is the type the message complains about, and we are throwing an instance of it on line 299 in ConnectionAttributes.cpp, but as far as I know, that shouldn't cause std::terminate() to be thrown, because:
- As far as I can tell, this isn't being thrown from a destructor during stack unwinding prompted by a prior exception (I added tracing into the exception type's constructor, and it's only called once before the crash)
- The current thread's stack has at least one frame with a
catch (Simba::Support::ErrorException&)
on it (and that same frame has acatch (...)
too, just to be sure), with probably others above it that should catch the exception.
What other reasons could there be for this? Am I missing something with my logic?
Given that it's only happening on Solaris, I'm thinking some issue in the runtime?
Edit: Some more details
uname -a
output on the machine:SunOS b3-sol10spr-01 5.10 Generic_147440-01 sun4v sparc sun4v
- We're building w/ GCC 4.9.2
- This is happening on a SPARC box, but we've had other issues related to exceptions on Solaris x86 before: Crash caused by invalid exception unwinding?
- Flags used during compilation of the driver:
-DSIZEOF_LONG_INT=4 -DSQL_WCHART_CONVERT -DHAVE_LONG_LONG -m32 -fPIC -pthread -Wall -Wno-unknown-pragmas -DSIMBA -D_REENTRANT -DCLUNIX -DNDEBUG -D_POSIX_PTHREAD_SEMANTICS -O0 -g -D_DEBUG
- Flags used during linking of the driver:
-DSIMBA -D_REENTRANT -m32 -fPIC -pthread -Wall -Wno-unknown-pragmas -O1 -g -shared -mimpure-text -static-libgcc -Wl,-M,exports_Solaris.map
(skipped a bunch of -lxxx)
LDD output for shared library (which houses the offending code)
libicudata_sb32.so.53 => /bamboo/Adhoc/ThirdParty/icu/53.1.x/solaris10sparc/gcc4_9/release32/lib/libicudata_sb32.so.53
libicui18n_sb32.so.53 => /bamboo/Adhoc/ThirdParty/icu/53.1.x/solaris10sparc/gcc4_9/release32/lib/libicui18n_sb32.so.53
libicuuc_sb32.so.53 => /bamboo/Adhoc/ThirdParty/icu/53.1.x/solaris10sparc/gcc4_9/release32/lib/libicuuc_sb32.so.53
libpthread.so.1 => /lib/libpthread.so.1
librt.so.1 => /lib/librt.so.1
libnsl.so.1 => /lib/libnsl.so.1
libsocket.so.1 => /lib/libsocket.so.1
libresolv.so.2 => /lib/libresolv.so.2
libxnet.so.1 => /lib/libxnet.so.1
libgssapi_krb5.so.2 => /opt/csw/lib/libgssapi_krb5.so.2
libkrb5.so.3 => /opt/csw/lib/libkrb5.so.3
libkrb5support.so.0 => /opt/csw/lib/libkrb5support.so.0
libk5crypto.so.3 => /opt/csw/lib/libk5crypto.so.3
libcom_err.so.3 => /opt/csw/lib/libcom_err.so.3
libm.so.2 => /lib/libm.so.2
libc.so.1 => /lib/libc.so.1
libgcc_s.so.1 => /opt/csw/lib/libgcc_s.so.1
libaio.so.1 => /lib/libaio.so.1
libmd.so.1 => /lib/libmd.so.1
libmp.so.2 => /lib/libmp.so.2
libscf.so.1 => /lib/libscf.so.1
libintl.so.8 => /opt/csw/lib/sparcv8/libintl.so.8
libdoor.so.1 => /lib/libdoor.so.1
libuutil.so.1 => /lib/libuutil.so.1
libgen.so.1 => /lib/libgen.so.1
libiconv.so.2 => /opt/csw/lib/sparcv8/libiconv.so.2
/lib/libm/libm_hwcap1.so.2
/platform/sun4v/lib/libc_psr.so.1
/platform/sun4v/lib/libmd_psr.so.1
LDD output for iodbctest (Not sure how this was built)
libdl.so.1 => /lib/libdl.so.1
libc.so.1 => /lib/libc.so.1
libm.so.2 => /lib/libm.so.2
/lib/libm/libm_hwcap1.so.2
/platform/sun4v/lib/libc_psr.so.1
Also, for someone who asked, if I set dbx to break on exceptions, I first break at this stack
(dbx) where -h
current thread: t@1
dbx: forward reference `Connection' not defined, see `help forwardref'
=>[1] __cxa_throw(obj = 0x13a268, tinfo = 0xfef2bf2c, dest = 0xfe204240 = &Simba::Support::ErrorException::~ErrorException()), line 65 in "eh_throw.cc"
[2] Simba::ODBC::ConnectionAttributes::SetAttribute(this = 0x12d5d0, in_attribute = <value unavailable>, in_value = 0x13a1e8, in_length = -3), line 299 in "ConnectionAttributes.cpp"
and then if I continue, it crashes at the location previously mentioned.
Edit 2: Here's some more stack frames when terminate is called:
(dbx) where -h
current thread: t@1
dbx: forward reference `Connection' not defined, see `help forwardref'
[1] __lwp_kill(0x0, 0x6, 0x0, 0x6, 0xffbffeff, 0x0), at 0xff2ce364
[2] raise(0x6, 0x0, 0x0, 0xff2ad954, 0xffffffff, 0x6), at 0xff267958
[3] abort(0x132cb0, 0x1, 0xff122f38, 0xedbe0, 0xff333558, 0x0), at 0xff242910
=>[4] __gnu_cxx::__verbose_terminate_handler(), line 95 in "vterminate.cc"
[5] __cxxabiv1::__terminate(handler = 0xff125098 = &__gnu_cxx::__verbose_terminate_handler()), line 47 in "eh_terminate.cc"
[6] std::terminate(), line 57 in "eh_terminate.cc"
[7] __cxa_throw(obj = 0x132cf8, tinfo = 0xfef2bf2c, dest = 0xfe204240 = &Simba::Support::ErrorException::~ErrorException()), line 87 in "eh_throw.cc"
[8] Simba::ODBC::ConnectionAttributes::SetAttribute(this = 0x12a060, in_attribute = <value unavailable>, in_value = 0x132c78, in_length = -3), line 299 in "ConnectionAttributes.cpp"
[9] Simba::ODBC::ConnectionState::SQLSetConnectAttr(this = <value unavailable>, in_connection = 0x129e08, Attribute = 1051, ValuePtr = 0x132c78, StringLength = -3), line 495 in "ConnectionState.cpp"
[10] Simba::ODBC::ConnectionState2::SQLSetConnectAttr(this = 0xd7a30, in_connection = 0x129e08, Attribute = 1051, ValuePtr = 0x132c78, StringLength = -3), line 526 in "ConnectionState2.cpp"
[11] Simba::ODBC::Connection::SQLSetConnectAttr(this = 0x129e08, Attribute = 1051, ValuePtr = 0x132c78, StringLength = -3), line 1746 in "Connection.cpp"
[12] Simba::ODBC::SQLSetConnectAttrTask<true>::DoSynchronously(, line 266 in "SQLSetConnectAttrTask.h"
[13] DoTask<Simba::ODBC::SQLSetConnectAttrTask<true> >(in_functionName = 0xfea68b10 "SQLSetConnectAttr", in_handle = 0x2, in_parameters = STRUCT), line 638 in "CInterface.cpp"
[14] SQLSetConnectAttrW(ConnectionHandle = 0x2, Attribute = 1051, Value = 0x132c78, StringLength = -3), line 4053 in "CInterface.cpp"
[15] _iodbcdm_SetConnectOption_init(0x9a568, 0x41b, 0x7b4a0, 0x41, 0x99e68, 0xffffefc8), at 0x23ae0
[16] _iodbcdm_driverload(0xffbfec10, 0xffbfec40, 0x9a568, 0x1, 0x0, 0x41), at 0x26458
[17] SQLDriverConnect_Internal(0x9a568, 0x0, 0xffbff590, 0xfffffffd, 0x98dc0, 0x1000), at 0x2a72c
[18] SQLDriverConnect(0x9a568, 0x0, 0xffbff590, 0xfffffffd, 0x98dc0, 0x1000), at 0x2b7e0
[19] ODBC_Connect(0xffbffb7b, 0x98c80, 0x0, 0x98c9e, 0xff3303ec, 0x4), at 0x16638
[20] main(0x2, 0xffbffa7c, 0xffbffa88, 0x99dc0, 0xff3700c0, 0x0), at 0x1837c
Also, I've confirmed that iodbctest was in fact built with gcc
Making all in samples
gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../include -I../include -DNO_FRAMEWORKS -DWITH_PTHREADS -D_REENTRANT -m32 -DODBCVER=0x0350 -DIODBC_BUILD=8120326 -MT iodbctest.o -MD -MP -MF .deps/iodbctest.Tpo -c -o iodbctest.o iodbctest.c
mv -f .deps/iodbctest.Tpo .deps/iodbctest.Po
/bin/bash ../libtool --tag=CC --mode=link gcc -std=gnu99 -m32 -DODBCVER=0x0350 -DIODBC_BUILD=8120326 -static -m32 -o iodbctest iodbctest.o ../iodbc/libiodbc.la
libtool: link: gcc -std=gnu99 -m32 -DODBCVER=0x0350 -DIODBC_BUILD=8120326 -m32 -o iodbctest iodbctest.o ../iodbc/.libs/libiodbc.a -ldl
Edit 3: Here's the link command for the .so
/opt/csw/gcc4/bin/g++ -DSIMBA -D_REENTRANT -m32 -fPIC -pthread -Wall -Wno-unknown-pragmas -lrt -O1 -g -shared API/ImpalaClient_solaris10sparc_gcc4_9_release32.cpp.o API/ImpalaTCLIServiceThreadSafeClient_solaris10sparc_gcc4_9_release32.cpp.o Core/ImpalaConnection_solaris10sparc_gcc4_9_release32.cpp.o Core/ImpalaDriver_solaris10sparc_gcc4_9_release32.cpp.o Core/ImpalaEnvironment_solaris10sparc_gcc4_9_release32.cpp.o Core/ImpalaStatement_solaris10sparc_gcc4_9_release32.cpp.o DataEngine/ImpalaAggregateFunction_solaris10sparc_gcc4_9_release32.cpp.o DataEngine/ImpalaCaseInsensitiveMetadataResult_solaris10sparc_gcc4_9_release32.cpp.o DataEngine/ImpalaDataEngine_solaris10sparc_gcc4_9_release32.cpp.o DataEngine/ImpalaDSIExtQueryExecutor_solaris10sparc_gcc4_9_release32.cpp.o DataEngine/ImpalaFetchThread_solaris10sparc_gcc4_9_release32.cpp.o DataEngine/ImpalaNativeQueryExecutor_solaris10sparc_gcc4_9_release32.cpp.o DataEngine/ImpalaPlaceholderTable_solaris10sparc_gcc4_9_release32.cpp.o DataEngine/ImpalaResultSet_solaris10sparc_gcc4_9_release32.cpp.o DataEngine/ImpalaScalarFunction_solaris10sparc_gcc4_9_release32.cpp.o DataEngine/Metadata/ImpalaColumnsMetadataSource_solaris10sparc_gcc4_9_release32.cpp.o DataEngine/Metadata/ImpalaMetadataHelper_solaris10sparc_gcc4_9_release32.cpp.o DataEngine/Metadata/ImpalaTablesMetadataSource_solaris10sparc_gcc4_9_release32.cpp.o DataEngine/Metadata/ImpalaTypeInfoMetadataSource_solaris10sparc_gcc4_9_release32.cpp.o QueryTranslation/ImpalaSQLGenerator_solaris10sparc_gcc4_9_release32.cpp.o Utilities/ImpalaTypeUtilities_solaris10sparc_gcc4_9_release32.cpp.o Utilities/ImpalaUtilities_solaris10sparc_gcc4_9_release32.cpp.o ../ThirdParty/ImpalaClients_Thrift_0_9_0/TCLIServiceClient/src/TCLIService_solaris10sparc_gcc4_9_release32.cpp.o ../ThirdParty/ImpalaClients_Thrift_0_9_0/TCLIServiceClient/src/TCLIService_constants_solaris10sparc_gcc4_9_release32.cpp.o ../ThirdParty/ImpalaClients_Thrift_0_9_0/TCLIServiceClient/src/TCLIService_types_solaris10sparc_gcc4_9_release32.cpp.o Main_Unix_solaris10sparc_gcc4_9_release32.cpp.o -mimpure-text -static-libgcc -L./solaris10sparc/gcc4_9/release32 -L/bamboo/bamboo-agent-home/xml-data/build-dir/ThirdParty/icu/53.1.x/solaris10sparc/gcc4_9/release32/lib -licudata_sb32 -licui18n_sb32 -licuuc_sb32 -lpthread -lm -Wl,-M,exports_Solaris.map -lrt -lnsl -lsocket -lresolv -lxnet -Wl,-M,exports_Solaris.map -Wl,-zallextract,/bamboo/bamboo-agent-home/xml-data/build-dir/SimbaEngine/Maintenance/10.1/Product/Lib/solaris10sparc/gcc4_9/release32/libSimbaODBC.a,/bamboo/bamboo-agent-home/xml-data/build-dir/SimbaEngine/Maintenance/10.1/Product/Lib/solaris10sparc/gcc4_9/release32/libSimbaDSI.a,/bamboo/bamboo-agent-home/xml-data/build-dir/SimbaEngine/Maintenance/10.1/Product/Lib/solaris10sparc/gcc4_9/release32/libSimbaSupport.a,/bamboo/bamboo-agent-home/xml-data/build-dir/SimbaEngine/Maintenance/10.1/Product/Lib/solaris10sparc/gcc4_9/release32/libAEProcessor.a,/bamboo/bamboo-agent-home/xml-data/build-dir/SimbaEngine/Maintenance/10.1/Product/Lib/solaris10sparc/gcc4_9/release32/libCore.a,/bamboo/bamboo-agent-home/xml-data/build-dir/SimbaEngine/Maintenance/10.1/Product/Lib/solaris10sparc/gcc4_9/release32/libDSIExt.a,/bamboo/bamboo-agent-home/xml-data/build-dir/SimbaEngine/Maintenance/10.1/Product/Lib/solaris10sparc/gcc4_9/release32/libExecutor.a,/bamboo/bamboo-agent-home/xml-data/build-dir/SimbaEngine/Maintenance/10.1/Product/Lib/solaris10sparc/gcc4_9/release32/libParser.a,/bamboo/bamboo-agent-home/xml-data/build-dir/Drivers/DriverShared/KerberosSupport/Maintenance/10.1/Lib/solaris10sparc/gcc4_9/release32/libKerberosSupport.a,/bamboo/bamboo-agent-home/xml-data/build-dir/Drivers/DriverShared/ThriftExtension/0.9.0/Maintenance/10.1/Lib/solaris10sparc/gcc4_9/release32/libThriftExtension.a,/bamboo/bamboo-agent-home/xml-data/build-dir/Drivers/DriverShared/DriverSupport/Maintenance/10.1/Lib/solaris10sparc/gcc4_9/release32/libDriverSupport.a -Wl,-zweakextract,/bamboo/bamboo-agent-home/xml-data/build-dir/ThirdParty/openssl/1.1.0/solaris10sparc/gcc4_9/release32/lib/libssl.a,/bamboo/bamboo-agent-home/xml-data/build-dir/ThirdParty/openssl/1.1.0/solaris10sparc/gcc4_9/release32/lib/libcrypto.a,/bamboo/bamboo-agent-home/xml-data/build-dir/Drivers/DriverShared/cyrus-sasl/2.1.26/saslGSSAPI/Maintenance/1.0/Binaries/solaris10sparc/gcc4_9/release32/lib/libsasl2.a,/bamboo/bamboo-agent-home/xml-data/build-dir/ThirdParty/boost/1.57.0_sb_icusb_53.1.x/solaris10sparc/gcc4_9/release32/lib/libboost_sb_date_time.a,/bamboo/bamboo-agent-home/xml-data/build-dir/ThirdParty/boost/1.57.0_sb_icusb_53.1.x/solaris10sparc/gcc4_9/release32/lib/libboost_sb_regex.a,/bamboo/bamboo-agent-home/xml-data/build-dir/ThirdParty/boost/1.57.0_sb_icusb_53.1.x/solaris10sparc/gcc4_9/release32/lib/libboost_sb_thread.a,/bamboo/bamboo-agent-home/xml-data/build-dir/ThirdParty/boost/1.57.0_sb_icusb_53.1.x/solaris10sparc/gcc4_9/release32/lib/libboost_sb_system.a,/bamboo/bamboo-agent-home/xml-data/build-dir/ThirdParty/thrift/0.9.0_openssl1.1.0_boost1.57.0_sb_icusb_53.1.x/solaris10sparc/gcc4_9/release32/lib/libthrift.a,/bamboo/bamboo-agent-home/xml-data/build-dir/ThirdParty/boost/1.57.0_sb_icusb_53.1.x/solaris10sparc/gcc4_9/release32/lib/libboost_sb_atomic.a,/bamboo/bamboo-agent-home/xml-data/build-dir/Drivers/DriverShared/cyrus-sasl/2.1.26/saslGSSAPI/Maintenance/1.0/Binaries/solaris10sparc/gcc4_9/release32/lib/sasl2/libgs2.a -L/bamboo/bamboo-agent-home/xml-data/build-dir/ThirdParty/MIT_Kerberos/1.14/solaris10sparc/gcc4_9/release32/lib -lgssapi_krb5 -lkrb5 -lkrb5support -lk5crypto -lcom_err -o ../Bin/solaris10sparc/gcc4_9/release32/libimpalaodbc_sb32.so