5

This is happening on a Solaris sparc machine:

-bash-3.2$ uname -a
SunOS b2s-sol10spr-1 5.10 Generic_147147-26 sun4v sparc sun4v

I'm seeing a strange issue with exception handling. Here's a log of a short debugging session:

GNU gdb (GDB) 7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "sparc-sun-solaris2.10".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./Touchstone_solaris64_iodbc...done.
(gdb) catch throw
Catchpoint 1 (throw)
(gdb) catch catch
Catchpoint 2 (catch)
(gdb) r
Starting program: /bamboo/mattheww/Touchstone_solaris64_iodbc -te Tests/Touchstone/SQL/SqlTestEnv_utf32.xml -ts Tests/Touchstone/SQL/SqlTestSuite.xml -o failure -rtn EXPECTEDFAILURES_QUERIESONLY-15
[Thread debugging using libthread_db enabled]
[New Thread 1 (LWP 1)]
Simba Test Verbose Log Started on Fri Aug 17 12:51:50 2018
Starting test run
---------------------------

Touchstone test utility for ODBC and OLE DB for OLAP
Version: 4.5.0.5 (64-bit)
Copyright (c) 2018 Simba Technologies Incorporated
__BUILD_INFO__    built at __BUILD_TIME__


getpid=22056



[Switching to Thread 1 (LWP 1)]
Catchpoint 1 (exception thrown), __cxxabiv1::__cxa_throw (obj=0x101ec52f0, tinfo=0xffffffff7b8d0138 <typeinfo for Simba::SQLEngine::SESqlErrorException>, dest=
    0xffffffff7a697850 <Simba::SQLEngine::SESqlErrorException::~SESqlErrorException()>) at /home/dam/mgar/pkg/gcc5/trunk/work/solaris10-sparc/build-isa-sparcv8plus/gcc-5.5.0/libstdc++-v3/libsupc++/eh_throw.cc:65
65      /home/dam/mgar/pkg/gcc5/trunk/work/solaris10-sparc/build-isa-sparcv8plus/gcc-5.5.0/libstdc++-v3/libsupc++/eh_throw.cc: No such file or directory.
(gdb) where
#0  __cxxabiv1::__cxa_throw (obj=0x101ec52f0, tinfo=0xffffffff7b8d0138 <typeinfo for Simba::SQLEngine::SESqlErrorException>, dest=0xffffffff7a697850 <Simba::SQLEngine::SESqlErrorException::~SESqlErrorException()>)
    at /home/dam/mgar/pkg/gcc5/trunk/work/solaris10-sparc/build-isa-sparcv8plus/gcc-5.5.0/libstdc++-v3/libsupc++/eh_throw.cc:65
#1  0xffffffff7a6f4188 in Simba::SQLEngine::AEScalarFnMetadataFactory::ValidateCotArgs (in_argument=0) at AEBuilder/Value/AEScalarFnMetadataFactory.cpp:2186
#2  0xffffffff7ab72d28 in Simba::SQLEngine::ETCotFn::RetrieveData (this=0x101db8ff0, io_dataRequest=...) at ETree/Value/ScalarFunctions/ETCotFn.cpp:37
#3  0xffffffff7a98c96c in Simba::SQLEngine::ETComparisonT<Simba::SQLEngine::ETEQFunctorT<double> >::GetLeftData (this=0x101db90f0) at Include/ETree/ETComparisonT.h:80
#4  0xffffffff7a976ac4 in Simba::SQLEngine::ETComparisonT<Simba::SQLEngine::ETEQFunctorT<double> >::Evaluate (this=0x101db90f0) at Include/ETree/ETComparisonT.h:104
#5  0xffffffff7a9c2510 in Simba::SQLEngine::ETSelect::DoMove (this=0x101ec53f0, in_moveRequest=...) at ETree/Relational/ETSelect.cpp:143
#6  0xffffffff7a8b4e18 in Simba::SQLEngine::ETRelationalExpr::Move (this=0x101ec53f0, in_moveRequest=...) at ../../../Include/SQLEngine/Executor/ETree/ETRelationalExpr.h:352
#7  0xffffffff7a9b8c90 in Simba::SQLEngine::ETProject::DoMove (this=0x101ec5450, in_moveRequest=...) at ETree/Relational/ETProject.cpp:143
#8  0xffffffff7a8b4e18 in Simba::SQLEngine::ETRelationalExpr::Move (this=0x101ec5450, in_moveRequest=...) at ../../../Include/SQLEngine/Executor/ETree/ETRelationalExpr.h:352
#9  0xffffffff7a8b4ae8 in Simba::SQLEngine::ETResultSet::Move (this=0x101dd6c40, in_direction=Simba::DSI::DSI_DIR_NEXT, in_offset=0) at ETResultSet.cpp:158
#10 0xffffffff7ad4f590 in Simba::ODBC::ForwardOnlyCursor::FetchRowset (this=0x101ec5650, in_orientation=1, in_offset=0, in_rowsetSize=1, in_ard=0x1014de6f0, in_rowStatusPtr=0x0, in_rowsProcessedPtr=0x0)
    at Cursor/ForwardOnlyCursor.cpp:329
#11 0xffffffff7adc0c88 in Simba::ODBC::QueryManager::FetchRowset (this=0x101eb4810, in_orientation=1, in_offset=0, in_rowsetSize=1, in_rowStatusPtr=0x0, in_rowsProcessedPtr=0x0) at QueryManager/QueryManager.cpp:87
#12 0xffffffff7adf93a0 in Simba::ODBC::StatementStateCursor::DoFetchScroll (this=0x101b12280, in_fetchOrientation=1, in_fetchOffset=0) at Statement/StatementStateCursor.cpp:823
#13 0xffffffff7ae06f18 in Simba::ODBC::StatementState5::SQLFetch (this=0x101b12280) at Statement/StatementState5.cpp:74
#14 0xffffffff7add6f20 in Simba::ODBC::Statement::SQLFetch (this=0x101e63c30) at Statement/Statement.cpp:986
#15 0xffffffff7acb1610 in Simba::ODBC::SQLFetchTask::DoSynchronously (in_stmt=..., in_params=...) at CInterface/SQLFetchTask.h:73
#16 0xffffffff7acc5884 in DoTask<Simba::ODBC::SQLFetchTask> (in_functionName=0xffffffff7af8fcb0 "SQLFetch", in_handle=0x3, in_parameters=...) at CInterface/CInterface.cpp:679
#17 0xffffffff7ac9ab3c in SQLFetch (StatementHandle=0x3) at CInterface/CInterface.cpp:1693
#18 0xffffffff7d32a1f4 in SQLFetch_Internal (hstmt=0x101b04fb0) at fetch.c:161
#19 0xffffffff7d32a560 in SQLFetch (hstmt=0x101b04fb0) at fetch.c:230
#20 0x00000001001f7454 in Simba::ODBCTest::Cli::SqlFetch (this=0x10143a3b0 <Simba::ODBCTest::Singleton<Simba::ODBCTest::Cli>::m_instance>, handle=0x101b04fb0) at SimbaODBCTestFramework/Cli/Cli.cpp:1109
#21 0x00000001002c9c98 in Simba::ODBCTest::Statement::SqlFetch (this=0x101e640a0, outcome=..., callerFile=0x100df8af0 "TestCases/SQLTests/ODBCXmlResultTestsBase.cpp", callerLine=828) at SimbaODBCTestFramework/ODBC/Statement.cpp:200
#22 0x0000000100c9ae98 in Simba::ODBCTest::ODBCXmlResultTestsBase::Pimpl::ValidateRows (this=0x1014e0640, test=..., xmlResult=0x1014df050) at TestCases/SQLTests/ODBCXmlResultTestsBase.cpp:828
#23 0x0000000100c96a94 in Simba::ODBCTest::ODBCXmlResultTestsBase::DoResultValidation (this=0x1018fc800, test=..., xmlFileName=..., expectedFileName=..., queryFileName=...) at TestCases/SQLTests/ODBCXmlResultTestsBase.cpp:504
#24 0x0000000100ca5ca4 in Simba::ODBCTest::ODBCXmlSimpleResultTestsBase::executeTest (this=0x1018fc800) at TestCases/SQLTests/ODBCXmlSimpleResultTestsBase.cpp:215
#25 0x0000000100ca8478 in Simba::Test::Case::runTest (this=0x1018fc800, runId=...) at TestFrameworkLibrary/SBTCase.cpp:180
#26 0x0000000100cb3b58 in Simba::Test::Engine::runTest (this=0xffffffff7ffff2a0, test=0x1018fc800) at TestFrameworkLibrary/SBTEngine.cpp:219
#27 0x0000000100cb3368 in Simba::Test::Engine::RunTests (this=0xffffffff7ffff2a0, testEnv=0x1014dfed0, loopCount=1) at TestFrameworkLibrary/SBTEngine.cpp:186
#28 0x00000001001f2090 in (anonymous namespace)::DoMain (argc=9, argv=0xffffffff7ffff898) at Main.cpp:557
#29 0x00000001001f256c in main (argc=9, argv=0xffffffff7ffff898) at Main.cpp:581
(gdb) cont
Continuing.
Catchpoint 1 (exception thrown), __cxxabiv1::__cxa_throw (obj=0x101ff73b0, tinfo=0x1013b9960 <typeinfo for Simba::Test::ValueMismatchException>, dest=0x100ce4e34 <Simba::Test::ValueMismatchException::~ValueMismatchException()>)
    at /home/dam/mgar/pkg/gcc5/trunk/work/solaris10-sparc/build-isa-sparcv8plus/gcc-5.5.0/libstdc++-v3/libsupc++/eh_throw.cc:65
65      in /home/dam/mgar/pkg/gcc5/trunk/work/solaris10-sparc/build-isa-sparcv8plus/gcc-5.5.0/libstdc++-v3/libsupc++/eh_throw.cc
(gdb) where
#0  __cxxabiv1::__cxa_throw (obj=0x101ff73b0, tinfo=0x1013b9960 <typeinfo for Simba::Test::ValueMismatchException>, dest=0x100ce4e34 <Simba::Test::ValueMismatchException::~ValueMismatchException()>)
    at /home/dam/mgar/pkg/gcc5/trunk/work/solaris10-sparc/build-isa-sparcv8plus/gcc-5.5.0/libstdc++-v3/libsupc++/eh_throw.cc:65
#1  0x0000000100ce4f28 in Simba::Test::ValueMismatchException::raise (this=0xffffffff7fffdd38) at TestFrameworkLibrary/Exceptions/SBTValueMismatchException.cpp:28
#2  0x000000010029e524 in Simba::ODBCTest::RaiseMismatchException (msg=...) at SimbaODBCTestFramework/Framework/VerifyValues.h:29
#3  0x000000010033c3d4 in Simba::ODBCTest::VerifyAndThrowComparator<Simba::ODBCTest::Comparator<short> > (comparator=..., msg=..., callerFile=0x100df8af0 "TestCases/SQLTests/ODBCXmlResultTestsBase.cpp", callerLine=888)
    at SimbaODBCTestFramework/Framework/VerifyValues.h:227
#4  0x000000010033c1b0 in Simba::ODBCTest::VerifyAndThrow<short> (expected=@0xffffffff7fffe34e: 1, actual=@0xffffffff7fffe396: 100, msg=..., callerFile=0x100df8af0 "TestCases/SQLTests/ODBCXmlResultTestsBase.cpp", callerLine=888)
    at SimbaODBCTestFramework/Framework/VerifyValues.h:251
#5  0x0000000100c9b450 in Simba::ODBCTest::ODBCXmlResultTestsBase::Pimpl::ValidateRows (this=0x1014e0640, test=..., xmlResult=0x1014df050) at TestCases/SQLTests/ODBCXmlResultTestsBase.cpp:888
#6  0x0000000100c96a94 in Simba::ODBCTest::ODBCXmlResultTestsBase::DoResultValidation (this=0x1018fc800, test=..., xmlFileName=..., expectedFileName=..., queryFileName=...) at TestCases/SQLTests/ODBCXmlResultTestsBase.cpp:504
#7  0x0000000100ca5ca4 in Simba::ODBCTest::ODBCXmlSimpleResultTestsBase::executeTest (this=0x1018fc800) at TestCases/SQLTests/ODBCXmlSimpleResultTestsBase.cpp:215
#8  0x0000000100ca8478 in Simba::Test::Case::runTest (this=0x1018fc800, runId=...) at TestFrameworkLibrary/SBTCase.cpp:180
#9  0x0000000100cb3b58 in Simba::Test::Engine::runTest (this=0xffffffff7ffff2a0, test=0x1018fc800) at TestFrameworkLibrary/SBTEngine.cpp:219
#10 0x0000000100cb3368 in Simba::Test::Engine::RunTests (this=0xffffffff7ffff2a0, testEnv=0x1014dfed0, loopCount=1) at TestFrameworkLibrary/SBTEngine.cpp:186
#11 0x00000001001f2090 in (anonymous namespace)::DoMain (argc=9, argv=0xffffffff7ffff898) at Main.cpp:557
#12 0x00000001001f256c in main (argc=9, argv=0xffffffff7ffff898) at Main.cpp:581
(gdb) cont
Continuing.
Catchpoint 2 (exception caught), __cxxabiv1::__cxa_begin_catch (exc_obj_in=0x101ff7390) at /home/dam/mgar/pkg/gcc5/trunk/work/solaris10-sparc/build-isa-sparcv8plus/gcc-5.5.0/libstdc++-v3/libsupc++/eh_catch.cc:41
41      /home/dam/mgar/pkg/gcc5/trunk/work/solaris10-sparc/build-isa-sparcv8plus/gcc-5.5.0/libstdc++-v3/libsupc++/eh_catch.cc: No such file or directory.
(gdb) where
#0  __cxxabiv1::__cxa_begin_catch (exc_obj_in=0x101ff7390) at /home/dam/mgar/pkg/gcc5/trunk/work/solaris10-sparc/build-isa-sparcv8plus/gcc-5.5.0/libstdc++-v3/libsupc++/eh_catch.cc:41
#1  0x0000000100ca8774 in Simba::Test::Case::runTest (this=0x1018fc800, runId=...) at TestFrameworkLibrary/SBTCase.cpp:183
#2  0x0000000100cb3b58 in Simba::Test::Engine::runTest (this=0xffffffff7ffff2a0, test=0x1018fc800) at TestFrameworkLibrary/SBTEngine.cpp:219
#3  0x0000000100cb3368 in Simba::Test::Engine::RunTests (this=0xffffffff7ffff2a0, testEnv=0x1014dfed0, loopCount=1) at TestFrameworkLibrary/SBTEngine.cpp:186
#4  0x00000001001f2090 in (anonymous namespace)::DoMain (argc=9, argv=0xffffffff7ffff898) at Main.cpp:557
#5  0x00000001001f256c in main (argc=9, argv=0xffffffff7ffff898) at Main.cpp:581
(gdb) info share
From                To                  Syms Read   Shared Object Library
0xffffffff7f606ca0  0xffffffff7f634270  Yes (*)     /usr/lib/sparcv9/ld.so.1
                                        Yes (*)     /bamboo/mattheww/icu-53.1.x/release64/lib//libicudata_sb64.so.53
0xffffffff7d8e00e8  0xffffffff7da07e0c  Yes (*)     /bamboo/mattheww/icu-53.1.x/release64/lib//libicui18n_sb64.so.53
0xffffffff7d56a798  0xffffffff7d62c31c  Yes (*)     /bamboo/mattheww/icu-53.1.x/release64/lib//libicuuc_sb64.so.53
                                        Yes (*)     /lib/64/libpthread.so.1
0xffffffff7d30f8b0  0xffffffff7d377994  Yes         /usr/local/odbc/dm/iodbc-3.52.8_gcc/64/debug/lib//libiodbc.so.2
0xffffffff7d104fe0  0xffffffff7d10cc20  Yes (*)     /lib/64/libsocket.so.1
0xffffffff7cf22340  0xffffffff7cf9e350  Yes (*)     /lib/64/libnsl.so.1
0xffffffff7cca9d38  0xffffffff7cd4a6a8  Yes         /opt/csw/lib/64/libstdc++.so.6
0xffffffff7ca0b080  0xffffffff7ca7b7a8  Yes (*)     /lib/64/libm.so.2
0xffffffff7c802d68  0xffffffff7c806000  Yes (*)     /lib/64/librt.so.1
0xffffffff7c602ef8  0xffffffff7c610028  Yes         /opt/csw/lib/64/libgcc_s.so.1
0xffffffff7c32ed60  0xffffffff7c3dd4b0  Yes (*)     /lib/64/libc.so.1
                                        Yes (*)     /lib/64/libdl.so.1
0xffffffff7bf02328  0xffffffff7bf09b38  Yes         /usr/sfw/lib/64/libgcc_s.so.1
0xffffffff7bd01e60  0xffffffff7bd076ac  Yes (*)     /lib/64/libaio.so.1
0xffffffff7bb008a0  0xffffffff7bb0c880  Yes (*)     /lib/64/libmd.so.1
0xffffffff7c000460  0xffffffff7c00137c  Yes (*)     /platform/sun4v/lib/sparcv9/libc_psr.so.1
0xffffffff79dece38  0xffffffff7ae0d3ec  Yes         /bamboo/mattheww/DRIVER
0xffffffff796009b8  0xffffffff7960c840  Yes (*)     /platform/sun4v/lib/sparcv9/libmd_psr.so.1
0xffffffff79400bb0  0xffffffff79402bb0  Yes (*)     /lib/64/libmp.so.2
0xffffffff79207690  0xffffffff79217630  Yes (*)     /lib/64/libscf.so.1
0xffffffff79001358  0xffffffff79002008  Yes (*)     /lib/64/libdoor.so.1
0xffffffff78e02518  0xffffffff78e06c58  Yes (*)     /lib/64/libuutil.so.1
0xffffffff78c01f08  0xffffffff78c06104  Yes (*)     /lib/64/libgen.so.1
0xffffffff78a03030  0xffffffff78a21078  Yes         /usr/local/odbc/dm/iodbc-3.52.8_gcc/64/debug/lib//libiodbcinst.so
(*): Shared library is missing debugging information.
(gdb) quit

After the first exception is thrown, frame 14 has a catch block for a base class of SESqlErrorException, so it should be caught there (and there are no intervening catch blocks that should). Yet, gdb's catch catch fails to break.

Also, the thread/process isn't killed; instead it seems like execution continues somehow (without causing any other problems) until we return into the main binary (frame 20 after the initial throw). I'm still not sure exactly where execution continues after the throw, will need to look into that later.

But, it seems that exception handling is working in the main binary, as we break when the main binary throws & catches an exception.

I don't have it in the log above, but I've noticed that the address of the instruction it breaks on when throwing the exception (i.e __cxxabiv1::__cxa_begin_catch) is actually slightly outside of the memory range that info share shows for libstdc++.so.6. How do I interpret that? I was looking at that, because earlier I had noticed that we had incorrectly built libicui18n_sb64.so.53 to be statically-linked with the C++ runtime, and when this issue occurred, the instruction doing the throw was (as far as I could tell, again it was a little outside of the info share range, but not in any other library's range) in libicui18n_sb64.so.53. I thought this was the source of the issue (throwing an exception using code from one copy of the runtime, so it wouldn't be caught by another copy), but fixing that library didn't make the issue go away.

Something else I noticed was that it claims that __cxxabiv1::__cxa_throw comes from /home/dam/mgar/pkg/gcc5/trunk/work/solaris10-sparc/build-isa-sparcv8plus/gcc-5.5.0/libstdc++-v3/libsupc++/eh_throw.cc:65, which makes me a bit suspicious since we compiled everything with gcc 4.9, but not sure if that's an issue.

Here's some information about the dependencies of all the libraries/binaries in the callstack:

Dependencies for /bamboo/mattheww/icu-53.1.x/release64/lib//libicui18n_sb64.so.53:
        libicuuc_sb64.so.53 =>   /bamboo/mattheww/icu-53.1.x/release64/lib//libicuuc_sb64.so.53
        libicudata_sb64.so.53 =>         /bamboo/mattheww/icu-53.1.x/release64/lib//libicudata_sb64.so.53
        libpthread.so.1 =>       /lib/64/libpthread.so.1
        libstdc++.so.6 =>        /opt/csw/lib/64/libstdc++.so.6
        libm.so.2 =>     /lib/64/libm.so.2
        librt.so.1 =>    /lib/64/librt.so.1
        libgcc_s.so.1 =>         /opt/csw/lib/64/libgcc_s.so.1
        libc.so.1 =>     /lib/64/libc.so.1
        libaio.so.1 =>   /lib/64/libaio.so.1
        libmd.so.1 =>    /lib/64/libmd.so.1
        /lib/sparcv9/../libm/sparcv9/libm_hwcap1.so.2
        /platform/sun4v/lib/sparcv9/libc_psr.so.1
        /platform/sun4v/lib/sparcv9/libmd_psr.so.1
Dependencies for /bamboo/mattheww/icu-53.1.x/release64/lib//libicuuc_sb64.so.53:
        libicudata_sb64.so.53 =>         /bamboo/mattheww/icu-53.1.x/release64/lib//libicudata_sb64.so.53
        libpthread.so.1 =>       /lib/64/libpthread.so.1
        libstdc++.so.6 =>        /opt/csw/lib/64/libstdc++.so.6
        libm.so.2 =>     /lib/64/libm.so.2
        librt.so.1 =>    /lib/64/librt.so.1
        libgcc_s.so.1 =>         /opt/csw/lib/64/libgcc_s.so.1
        libc.so.1 =>     /lib/64/libc.so.1
        libaio.so.1 =>   /lib/64/libaio.so.1
        libmd.so.1 =>    /lib/64/libmd.so.1
        /lib/sparcv9/../libm/sparcv9/libm_hwcap1.so.2
        /platform/sun4v/lib/sparcv9/libc_psr.so.1
        /platform/sun4v/lib/sparcv9/libmd_psr.so.1
Dependencies for /usr/local/odbc/dm/iodbc-3.52.8_gcc/64/debug/lib//libiodbc.so.2:
        libdl.so.1 =>    /lib/64/libdl.so.1
        libc.so.1 =>     /lib/64/libc.so.1
        libgcc_s.so.1 =>         /usr/sfw/lib/64/libgcc_s.so.1
        libm.so.2 =>     /lib/64/libm.so.2
        /lib/sparcv9/../libm/sparcv9/libm_hwcap1.so.2
        /platform/sun4v/lib/sparcv9/libc_psr.so.1
Dependencies for /bamboo/mattheww/DRIVER:
        libstdc++.so.6 =>        /opt/csw/lib/64/libstdc++.so.6
        libicudata_sb64.so.53 =>         /bamboo/mattheww/icu-53.1.x/release64/lib//libicudata_sb64.so.53
        libicui18n_sb64.so.53 =>         /bamboo/mattheww/icu-53.1.x/release64/lib//libicui18n_sb64.so.53
        libicuuc_sb64.so.53 =>   /bamboo/mattheww/icu-53.1.x/release64/lib//libicuuc_sb64.so.53
        libpthread.so.1 =>       /lib/64/libpthread.so.1
        libsocket.so.1 =>        /lib/64/libsocket.so.1
        libnsl.so.1 =>   /lib/64/libnsl.so.1
        libm.so.2 =>     /lib/64/libm.so.2
        librt.so.1 =>    /lib/64/librt.so.1
        libgcc_s.so.1 =>         /opt/csw/lib/64/libgcc_s.so.1
        libc.so.1 =>     /lib/64/libc.so.1
        libmp.so.2 =>    /lib/64/libmp.so.2
        libmd.so.1 =>    /lib/64/libmd.so.1
        libscf.so.1 =>   /lib/64/libscf.so.1
        libaio.so.1 =>   /lib/64/libaio.so.1
        libdoor.so.1 =>  /lib/64/libdoor.so.1
        libuutil.so.1 =>         /lib/64/libuutil.so.1
        libgen.so.1 =>   /lib/64/libgen.so.1
        /lib/sparcv9/../libm/sparcv9/libm_hwcap1.so.2
        /platform/sun4v/lib/sparcv9/libc_psr.so.1
        /platform/sun4v/lib/sparcv9/libmd_psr.so.1
Dependencies for ./Touchstone_solaris64_iodbc:
        libicudata_sb64.so.53 =>         /bamboo/mattheww/icu-53.1.x/release64/lib//libicudata_sb64.so.53
        libicui18n_sb64.so.53 =>         /bamboo/mattheww/icu-53.1.x/release64/lib//libicui18n_sb64.so.53
        libicuuc_sb64.so.53 =>   /bamboo/mattheww/icu-53.1.x/release64/lib//libicuuc_sb64.so.53
        libpthread.so.1 =>       /lib/64/libpthread.so.1
        libiodbc.so.2 =>         /usr/local/odbc/dm/iodbc-3.52.8_gcc/64/debug/lib//libiodbc.so.2
        libsocket.so.1 =>        /lib/64/libsocket.so.1
        libnsl.so.1 =>   /lib/64/libnsl.so.1
        libstdc++.so.6 =>        /opt/csw/lib/64/libstdc++.so.6
        libm.so.2 =>     /lib/64/libm.so.2
        librt.so.1 =>    /lib/64/librt.so.1
        libgcc_s.so.1 =>         /opt/csw/lib/64/libgcc_s.so.1
        libc.so.1 =>     /lib/64/libc.so.1
        libdl.so.1 =>    /lib/64/libdl.so.1
        libgcc_s.so.1 =>         /usr/sfw/lib/64/libgcc_s.so.1
        libmp.so.2 =>    /lib/64/libmp.so.2
        libmd.so.1 =>    /lib/64/libmd.so.1
        libscf.so.1 =>   /lib/64/libscf.so.1
        libaio.so.1 =>   /lib/64/libaio.so.1
        libdoor.so.1 =>  /lib/64/libdoor.so.1
        libuutil.so.1 =>         /lib/64/libuutil.so.1
        libgen.so.1 =>   /lib/64/libgen.so.1
        /lib/sparcv9/../libm/sparcv9/libm_hwcap1.so.2
        /platform/sun4v/lib/sparcv9/libc_psr.so.1
        /platform/sun4v/lib/sparcv9/libmd_psr.so.1

They all depend on /opt/csw/lib/64/libstdc++.so.6 & /opt/csw/lib/64/libgcc_s.so.1, I'm not sure if any other libraries would be relevant w.r.t. exception handling? (I've read about/seen issues when mixing gcc & 'native' runtimes, but I don't think that's happening here?)

Here's how the library (/bamboo/mattheww/DRIVER) was built:

Example command for compiling one of the .cpp files (I removed a bunch of include directories)

/opt/csw/gcc4/bin/g++ -DSIZEOF_LONG_INT=8 -DSQL_WCHART_CONVERT -DHAVE_MEMMOVE -m64 -fPIC -pthread  -Wall -Wno-unknown-pragmas -DSIMBA -D_REENTRANT -DCLUNIX -DNDEBUG -D_POSIX_PTHREAD_SEMANTICS  -O0 -g -D_DEBUG  -c Common/QSTableMetadataFile.cpp -o Common/QSTableMetadataFile_solaris10sparc_gcc4_9_debug64.cpp.o

Command used to link (removed a bunch of .o files)

/opt/csw/gcc4/bin/g++ -DSIMBA -D_REENTRANT -m64 -fPIC -pthread  -Wall -Wno-unknown-pragmas  -lrt   -O0 -g  -shared -L/bamboo/bamboo-agent-home/xml-data/build-dir/ThirdParty/icu/53.1.x/solaris10sparc/gcc4_9/release64/lib -lstdc++ -licudata_sb64 -licui18n_sb64 -licuuc_sb64 -lpthread -lm -lsocket -lnsl -Wl,-M,exports_SunOS.map -Wl,-zallextract,/bamboo/bamboo-agent-home/xml-data/build-dir/SimbaEngine/Maintenance/10.1/Product/Lib/solaris10sparc/gcc4_9/debug64/libSimbaDSI.a,/bamboo/bamboo-agent-home/xml-data/build-dir/SimbaEngine/Maintenance/10.1/Product/Lib/solaris10sparc/gcc4_9/debug64/libSimbaSupport.a,/bamboo/bamboo-agent-home/xml-data/build-dir/SimbaEngine/Maintenance/10.1/Product/Lib/solaris10sparc/gcc4_9/debug64/libAEProcessor.a,/bamboo/bamboo-agent-home/xml-data/build-dir/SimbaEngine/Maintenance/10.1/Product/Lib/solaris10sparc/gcc4_9/debug64/libCore.a,/bamboo/bamboo-agent-home/xml-data/build-dir/SimbaEngine/Maintenance/10.1/Product/Lib/solaris10sparc/gcc4_9/debug64/libDSIExt.a,/bamboo/bamboo-agent-home/xml-data/build-dir/SimbaEngine/Maintenance/10.1/Product/Lib/solaris10sparc/gcc4_9/debug64/libExecutor.a,/bamboo/bamboo-agent-home/xml-data/build-dir/SimbaEngine/Maintenance/10.1/Product/Lib/solaris10sparc/gcc4_9/debug64/libParser.a,/bamboo/bamboo-agent-home/xml-data/build-dir/SimbaEngine/Maintenance/10.1/Product/Lib/solaris10sparc/gcc4_9/debug64/libSimbaODBC.a -Wl,-zweakextract  -o ../Bin/solaris10sparc/gcc4_9/debug64/libQuickstart64.so

Another thing to mention is that this happens in 64-bit, but not 32-bit. Originally, the 32-bit version of libicui18n_sb64.so.53 had been dynamically-linked to the runtime, while libicui18n_sb64.so.53 hadn't, so I thought that difference was the cause, but it didn't change the observed behaviour.

edit:

(gdb) where
#0  __cxxabiv1::__cxa_throw (obj=0x101ec52f0, tinfo=0xffffffff7b8d0138 <typeinfo for Simba::SQLEngine::SESqlErrorException>, dest=0xffffffff7a697850 <Simba::SQLEngine::SESqlErrorException::~SESqlErrorException()>)
    at /home/dam/mgar/pkg/gcc5/trunk/work/solaris10-sparc/build-isa-sparcv8plus/gcc-5.5.0/libstdc++-v3/libsupc++/eh_throw.cc:65
#1  0xffffffff7a6f4188 in Simba::SQLEngine::AEScalarFnMetadataFactory::ValidateCotArgs (in_argument=0) at AEBuilder/Value/AEScalarFnMetadataFactory.cpp:2186
#2  0xffffffff7ab72d28 in Simba::SQLEngine::ETCotFn::RetrieveData (this=0x101db8ff0, io_dataRequest=...) at ETree/Value/ScalarFunctions/ETCotFn.cpp:37
#3  0xffffffff7a98c96c in Simba::SQLEngine::ETComparisonT<Simba::SQLEngine::ETEQFunctorT<double> >::GetLeftData (this=0x101db90f0) at Include/ETree/ETComparisonT.h:80
#4  0xffffffff7a976ac4 in Simba::SQLEngine::ETComparisonT<Simba::SQLEngine::ETEQFunctorT<double> >::Evaluate (this=0x101db90f0) at Include/ETree/ETComparisonT.h:104
#5  0xffffffff7a9c2510 in Simba::SQLEngine::ETSelect::DoMove (this=0x101ec53f0, in_moveRequest=...) at ETree/Relational/ETSelect.cpp:143
#6  0xffffffff7a8b4e18 in Simba::SQLEngine::ETRelationalExpr::Move (this=0x101ec53f0, in_moveRequest=...) at ../../../Include/SQLEngine/Executor/ETree/ETRelationalExpr.h:352
#7  0xffffffff7a9b8c90 in Simba::SQLEngine::ETProject::DoMove (this=0x101ec5450, in_moveRequest=...) at ETree/Relational/ETProject.cpp:143
#8  0xffffffff7a8b4e18 in Simba::SQLEngine::ETRelationalExpr::Move (this=0x101ec5450, in_moveRequest=...) at ../../../Include/SQLEngine/Executor/ETree/ETRelationalExpr.h:352
#9  0xffffffff7a8b4ae8 in Simba::SQLEngine::ETResultSet::Move (this=0x101dd6c40, in_direction=Simba::DSI::DSI_DIR_NEXT, in_offset=0) at ETResultSet.cpp:158
<Snipped as this post is now too long>
(gdb) fin
Run till exit from #0  __cxxabiv1::__cxa_throw (obj=0x101ec52f0, tinfo=0xffffffff7b8d0138 <typeinfo for Simba::SQLEngine::SESqlErrorException>, dest=0xffffffff7a697850 <Simba::SQLEngine::SESqlErrorException::~SESqlErrorException()>)
    at /home/dam/mgar/pkg/gcc5/trunk/work/solaris10-sparc/build-isa-sparcv8plus/gcc-5.5.0/libstdc++-v3/libsupc++/eh_throw.cc:65
0xffffffff7a8b4e38 in Simba::SQLEngine::ETRelationalExpr::Move (this=0x101ec5450, in_moveRequest=...) at ../../../Include/SQLEngine/Executor/ETree/ETRelationalExpr.h:357
357     ../../../Include/SQLEngine/Executor/ETree/ETRelationalExpr.h: No such file or directory.
(gdb) where
#0  0xffffffff7a8b4e38 in Simba::SQLEngine::ETRelationalExpr::Move (this=0x101ec5450, in_moveRequest=...) at ../../../Include/SQLEngine/Executor/ETree/ETRelationalExpr.h:357
#1  0xffffffff7a8b4ae8 in Simba::SQLEngine::ETResultSet::Move (this=0x101dd6c40, in_direction=Simba::DSI::DSI_DIR_NEXT, in_offset=0) at ETResultSet.cpp:158
<Snipped as this post is now too long>
(gdb)

I'm not sure what exactly it means to do a fin in GDB when the exception handling machinery is about to unwind the stack, but it's back in the non-exceptional flow of code, without the exception being caught...

Edit: I've uploaded a zip file containing two binaries, one that works and one that doesn't (when I rebuild myself, it works, when our automated build system does, it doesn't): https://simba.app.box.com/s/qzo8735h9nzyv6t4x2lhzze896v361t1

You can reproduce the issue with iodbctest using a connection string like

DRIVER=<path to driver>;DBF=<path to DBF directory>

and executing the query

select dbl_zero from trig where COT(dbl_zero) = dbl_zero

Also, I dissassembled both the function throwing the exception, and the function which should catch that exception, in both binaries, and they were they same (other than some constants which I assume corresponded to different addresses).

Edit: I've created logs of when the test is run, with both working & broken binaries, using LD_DEBUG=bindings: https://simba.box.com/s/w5y246hb4ydm97krjeyjykc90zvjqfo3 I've noticed a few differences, but not sure exactly how to interpret them

Bwmat
  • 4,314
  • 3
  • 27
  • 42
  • Have you attempted to recreate this in a super-simple "hello world" type program, where sharing the details (and the code!) would be easier? – Steve Aug 17 '18 at 23:53
  • Not yet, the fact that I can't reproduce this when I build myself (rather than using our build system, which uses the same makefiles) deterred me a bit – Bwmat Aug 17 '18 at 23:58
  • Didn't check if the build flags end up being different when I did that though... – Bwmat Aug 17 '18 at 23:59
  • 2
    *Something else I noticed was that it claims that __cxxabiv1::__cxa_throw comes from /home/dam/mgar/pkg/gcc5/trunk/work/solaris10-sparc/build-isa-sparcv8plus/gcc-5.5.0/libstdc++-v3/libsupc++/eh_throw.cc:65, which makes me a bit suspicious since we compiled everything with gcc 4.9, but not sure if that's an issue.* Does that show up in the build you did yourself, or just in the ones from your build system? You're right to be suspicious of that. – Andrew Henle Aug 18 '18 at 13:09
  • I haven't had the chance to check what happens in the version I build myself yet, but I not think that the reason it was talking about 5.5 was just that the runtime on this machine was built for that, _but_, it should be forwards compatible, right? – Bwmat Aug 21 '18 at 00:53
  • @Bwmat *it should be forwards compatible* That doesn't mean it's going to be OK use 4.9 to compile against a 5.5 runtime. That would be backwards compatibility. – Andrew Henle Aug 22 '18 at 09:36
  • I think it's building using the 4.9 runtime (the dependency is on libstdc++.so.6, which resolves to libstdc++.so.6.21 for gcc 5.1+ and libstdc++.so.6.20 for 4.9) – Bwmat Aug 22 '18 at 18:24
  • And it looks like the build flags are the same – Bwmat Aug 22 '18 at 18:24
  • 1
    Could you try the same using mdb instead of gdb? I've noticed that mdb behaves rationally even in cases where gdb does not. – Mustafa Ozturk Aug 22 '18 at 21:32
  • @MustafaOzturk Can mdb set a breakpoint for when a C++ exception is thrown/caught? Couldn't find a way in my admittedly quick search (never used it before). Anyway, I don't think the weird behaviour is caused by the debugger, I have logging that confirms the exception is thrown, but that execution continues before it should have been caught. – Bwmat Aug 23 '18 at 04:16
  • This might be of some help: https://stackoverflow.com/questions/10454099/mdbs-substitute-for-gdbs-catch-throw It looks like you're dealing with either a mismatch somewhere in your C++ runtime, or you've found a compiler or runtime bug. I'd recompile everything with a known-to-be-consistent compiler/runtime on a known-to-be good system. And I mean everything that gets compiled - make sure it's **all** done on this known-good system. Yes, it's a tedious job, but from what you've posted so far you haven't eliminated the possibility of a C++ runtime mismatch from somewhere in your process. – Andrew Henle Aug 23 '18 at 09:44
  • I'd also install [the latest Oracle Developer Studio compilers](http://www.oracle.com/technetwork/server-storage/developerstudio/overview/index.html) The `dbx` debugger there is a lot easier to use than `mdb`, and is quite likely to be able to handle exceptions better than `mdb`. See https://docs.oracle.com/cd/E60778_01/html/E60748/blald.html#scrolltoc I'd also make sure your Solaris systems are patched and up-to-date. You may be seeing a C++ runtime bug. – Andrew Henle Aug 23 '18 at 09:50
  • @Bwmat it is possible. This is what I did to set a breakpoint in the exp throw function of libCrun `::bp libCrun.so.1\`__1cG__CrunIex_throw6Fpvpkn0AQstatic_type_info_pF1_v_v_` – Mustafa Ozturk Aug 23 '18 at 13:40
  • Did you try debugging the library loading process using `LD_DEBUG`? `LD_DEBUG=libs /bamboo/mattheww/Touchstone_solaris64_iodbc blah blah blah` will list all of the libraries loaded and why they are being loaded. You can identify what is causing a suspect library being linked into your app. See this link: https://docs.oracle.com/cd/E26502_01/html/E26507/chapter3-31.html – Mustafa Ozturk Aug 23 '18 at 13:45
  • Thanks for the LD_DEBUG tip, but I don't see anything different between the version of the library that works & the one that doesn't in that output unfortunately. – Bwmat Aug 23 '18 at 16:44
  • @Bwmat, I'd also debug symbol binding using the same utility. If that doesn't work, I would try to recreate the issue using by writing purpose built code to as per Steve's suggestion. You could submit that as a bug to the gcc devs. – Mustafa Ozturk Aug 23 '18 at 20:45

0 Answers0