Crash caused by invalid exception unwinding?

Question

I'm debugging a crash which occurs in one of our tests. We have an ODBC driver .so written in C++, which is being tested via iODBC using our test tool (called 'Touchstone'), which is also written in C++.

I've compiled all three on Solaris 10 (x86) in 64-bit mode using the Oracle Solaris Studio 12.4 (NOT GCC).

The crash only occurs when Touchstone is built in release mode, so I've spent a fair bit of time stepping through assembly in dbx, and what seems to happen is the following:

The 'this' pointer of the 'statement' object is stored into r14
We call into SQLColAttributes in the driver manager (DM), which calls into SQLColAttributeW in the driver.
Before returning into the DM, it spills r14 to the stack in one of SQLColAttributeW's callees.
Farther down the stack, an exception is thrown (which eventually is caught, before leaving the driver and returning into the DM), which seems to 'mess up' during stack unwinding, so that once I finally return into touchstone, r14 is garbage. Touchstone attempts to retrieve a member from the statement object and very quickly crashes with a SEGV.

One of the first things I did when I got to #3 was to put a hardware write watch on the stack where the register was getting spilled to, but it doesn't get hit until after returning into touchstone, after the register has already been corrupted.

Then I noticed that the code which seem to cause the corruption was throwing an exception, and remembered this, which I have encountered before, when using Touchstone built with solaris studio use a driver built with gcc, so I recompiled iODBC with solaris studio, and running ldd shows no dependency on it anymore, but it's still crashing in the same way.

I've also tried the workaround suggested in that article (LD_PRELOAD=/usr/sfw/lib/amd64/libgcc_s.so), but that didn't change anything either.

I was also able to compile valgrind, and it seems to support my hypothesis:

-bash-4.1$ $VG --tool=memcheck $TC -te ApiTestEnv_utf32.xml -ts ApiTestSuite.xml -o crash -rts COLATTRIBUTETESTS
==900== Memcheck, a memory error detector
==900== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==900== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==900== Command: /export/home/bamboo/Touchstone -te ApiTestEnv_utf32.xml -ts ApiTestSuite.xml -o crash -rts COLATTRIBUTETESTS
==900==
Simba Test Verbose Log Started on Thu Jul 14 11:35:25 2016

Touchstone test utility for ODBC and OLE DB for OLAP
Version: 4.5.0.5 (64-bit)
Copyright (c) 2012 Simba Technologies Incorporated

Starting test run
---------------------------
        API Tests: COLATTRIBUTETESTS: SQLCOLATTRIBUTES_ERROR (1)==900== Invalid read of size 8
==900==    at 0x7FE3BD7D2: _Unw_jmp (in /lib/amd64/libc.so.1)
==900==    by 0x7F94FF6D2: __1cFSimbaHSupportbAIniFileConfigurationReaderLOpenIniFile6Mrkn0BNsimba_wstring_5_pn0BITextFile__ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F94FEBC3: __1cFSimbaHSupportbAIniFileConfigurationReaderRLoadConfiguration6MrnDstdDmap4n0BNsimba_wstring_n0DDmap4n0E_n0BHVariant_n0EZCaseInsensitiveComparator_n0DJallocator4n0DEpair4Ck4n0F_______n0G_n0DJallocator4n0DEpai
r4C5n0J_______r58p5_b_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F94FEA51: __1cFSimbaHSupportbAIniFileConfigurationReaderRLoadConfiguration6MrnDstdDmap4n0BNsimba_wstring_n0BHVariant_n0EZCaseInsensitiveComparator_n0DJallocator4n0DEpair4Ck4n0F_______r58p5b_b_ (in /export/home/bamboo/sol
-crash/libInternalTest_debug.so)
==900==    by 0x7F950CD97: __1cFSimbaHSupportSSimbaSettingReaderUInternal_ReadSetting6MrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____4_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F950C2C6: __1cFSimbaHSupportSSimbaSettingReaderLReadSetting6FrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____4_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F950A2DC: __1cFSimbaHSupportSSimbaSettingReaderSGetAppCharEncoding6F_n0BMEncodingType__ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F8D8E29C: __1cFSimbaDDSIJDSIDriverYSetDefaultPropertyValues6M_v_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F8D8D787: __1cFSimbaDDSIJDSIDriver2t6M_v_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F7B57748: __1cFSimbaMInternalTestIITDriver2t6M_v_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F86D7330: __1cFSimbaDDSIQDSIDriverFactory6FrL_pn0BHIDriver__ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F8E17FEB: __1cFSimbaDDSIWSharedSingletonManagerKInitialize6Fb_v_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==  Address 0x37fe5ba8 is on thread 1's stack
==900==  664 bytes below stack pointer
==900==
==900== Invalid read of size 8
==900==    at 0x7FE3BD7D2: _Unw_jmp (in /lib/amd64/libc.so.1)
==900==    by 0x7F94FEBC3: __1cFSimbaHSupportbAIniFileConfigurationReaderRLoadConfiguration6MrnDstdDmap4n0BNsimba_wstring_n0DDmap4n0E_n0BHVariant_n0EZCaseInsensitiveComparator_n0DJallocator4n0DEpair4Ck4n0F_______n0G_n0DJallocator4n0DEpai
r4C5n0J_______r58p5_b_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F94FEA51: __1cFSimbaHSupportbAIniFileConfigurationReaderRLoadConfiguration6MrnDstdDmap4n0BNsimba_wstring_n0BHVariant_n0EZCaseInsensitiveComparator_n0DJallocator4n0DEpair4Ck4n0F_______r58p5b_b_ (in /export/home/bamboo/sol
-crash/libInternalTest_debug.so)
==900==    by 0x7F950CD97: __1cFSimbaHSupportSSimbaSettingReaderUInternal_ReadSetting6MrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____4_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F950C2C6: __1cFSimbaHSupportSSimbaSettingReaderLReadSetting6FrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____4_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F950A2DC: __1cFSimbaHSupportSSimbaSettingReaderSGetAppCharEncoding6F_n0BMEncodingType__ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F8D8E29C: __1cFSimbaDDSIJDSIDriverYSetDefaultPropertyValues6M_v_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F8D8D787: __1cFSimbaDDSIJDSIDriver2t6M_v_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F7B57748: __1cFSimbaMInternalTestIITDriver2t6M_v_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F86D7330: __1cFSimbaDDSIQDSIDriverFactory6FrL_pn0BHIDriver__ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F8E17FEB: __1cFSimbaDDSIWSharedSingletonManagerKInitialize6Fb_v_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FA0644BD: __1cFSimbaEODBCGDriverUInitializeSingletons6M_v_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==  Address 0x37fe5bd8 is on thread 1's stack
==900==  856 bytes below stack pointer
==900==
==900== Invalid read of size 8
==900==    at 0x7FE3BD7D2: _Unw_jmp (in /lib/amd64/libc.so.1)
==900==    by 0x7F7B58945: __1cFSimbaMInternalTestIITDriverbAInitializeUnicodeStringMap6M_v_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F7B57C96: __1cFSimbaMInternalTestIITDriverRCreateEnvironment6M_pn0ADDSIMIEnvironment__ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FA06087D: __1cFSimbaEODBCGDriverRCreateEnvironment6Mppv_h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9EB138F: SQLAllocHandle (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FE21BE90: _iodbcdm_driverload (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE21FDB0: SQLDriverConnect_Internal (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE220DE3: SQLDriverConnectW (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0xA69132: __1cFSimbaIODBCTestDCliRSqlDriverConnectW6Mpv3pwh4hphH_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAD9CDB: __1cFSimbaIODBCTestKConnectionRSqlDriverConnectW6MpvpkwhpwhphHrkn0BHOutcome_pkci_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xABC586: __1cFSimbaIODBCTestRConnectionFactorySMakeConnectionInC46Fpn0BLEnvironment_rkn0BHODBCStr__pn0BKConnection__ (in /export/home/bamboo/Touchstone)
==900==    by 0xAD2F39: __1cFSimbaIODBCTestSOdbcTestCaseBaseS1MexecuteSetup6M_b_ (in /export/home/bamboo/Touchstone)
==900==  Address 0x37fe6798 is on thread 1's stack
==900==  664 bytes below stack pointer
==900==
==900== Invalid read of size 8
==900==    at 0x7FE3BD7D2: _Unw_jmp (in /lib/amd64/libc.so.1)
==900==    by 0x7F7B57C96: __1cFSimbaMInternalTestIITDriverRCreateEnvironment6M_pn0ADDSIMIEnvironment__ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FA06087D: __1cFSimbaEODBCGDriverRCreateEnvironment6Mppv_h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9EB138F: SQLAllocHandle (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FE21BE90: _iodbcdm_driverload (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE21FDB0: SQLDriverConnect_Internal (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE220DE3: SQLDriverConnectW (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0xA69132: __1cFSimbaIODBCTestDCliRSqlDriverConnectW6Mpv3pwh4hphH_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAD9CDB: __1cFSimbaIODBCTestKConnectionRSqlDriverConnectW6MpvpkwhpwhphHrkn0BHOutcome_pkci_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xABC586: __1cFSimbaIODBCTestRConnectionFactorySMakeConnectionInC46Fpn0BLEnvironment_rkn0BHODBCStr__pn0BKConnection__ (in /export/home/bamboo/Touchstone)
==900==    by 0xAD2F39: __1cFSimbaIODBCTestSOdbcTestCaseBaseS1MexecuteSetup6M_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD228AB: __1c9uX__unnamed_Aj63VIoYhXFiiQColAttributeBaseMexecuteSetup6M_b_ (in /export/home/bamboo/Touchstone)
==900==  Address 0x37fe67c8 is on thread 1's stack
==900==  856 bytes below stack pointer
==900==
==900== Invalid read of size 8
==900==    at 0x7FE3BD7D2: _Unw_jmp (in /lib/amd64/libc.so.1)
==900==    by 0x7FA039B3D: __1cFSimbaEODBCKDescriptorOGetHeaderField6kMhpvpi_v_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FA11C104: __1cFSimbaEODBCOStatementStateQSQdDLNumResultCols6Mph_v_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FA10903D: __1cFSimbaEODBCJStatementQSQdDLNumResultCols6Mph_h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9F52DE0: __1cGDoTask4nFSimbaEODBCUSQdDLNumResultColsTask__6Fpkcpvrn7TAOTaskParameters__h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9ED63DB: SQLNumResultCols (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FE23C4E1: _iodbcdm_NumResultCols (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE223169: _iodbcdm_do_cursoropen (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE224295: SQLExecDirect_Internal (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE224537: SQLExecDirect (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0xA69B5D: __1cFSimbaIODBCTestDCliNSqlExecDirect6MpvpCi_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE39BA: __1cFSimbaIODBCTestJStatementNSqlExecDirect6MpkCirkn0BHOutcome_pkci_h_ (in /export/home/bamboo/Touchstone)
==900==  Address 0x37fee258 is on thread 1's stack
==900==  664 bytes below stack pointer
==900==
==900== Invalid read of size 8
==900==    at 0x7FE3BD7D2: _Unw_jmp (in /lib/amd64/libc.so.1)
==900==    by 0x7FA10903D: __1cFSimbaEODBCJStatementQSQdDLNumResultCols6Mph_h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9F52DE0: __1cGDoTask4nFSimbaEODBCUSQdDLNumResultColsTask__6Fpkcpvrn7TAOTaskParameters__h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9ED63DB: SQLNumResultCols (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FE23C4E1: _iodbcdm_NumResultCols (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE223169: _iodbcdm_do_cursoropen (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE224295: SQLExecDirect_Internal (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE224537: SQLExecDirect (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0xA69B5D: __1cFSimbaIODBCTestDCliNSqlExecDirect6MpvpCi_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE39BA: __1cFSimbaIODBCTestJStatementNSqlExecDirect6MpkCirkn0BHOutcome_pkci_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD205EA: __1c9uX__unnamed_Aj63VIoYhXFiiWSQdDLCOLATTRIBUTES_ERRORLexecuteTest6M_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0x12F954D: __1cFSimbaETestECaseHrunTest6MrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____n0CLTEST_STATUS__ (in /export/home/bamboo/Touchstone)
==900==  Address 0x37fee288 is on thread 1's stack
==900==  1128 bytes below stack pointer
==900==
==900== Invalid read of size 8
==900==    at 0x7FE3BD7D2: _Unw_jmp (in /lib/amd64/libc.so.1)
==900==    by 0x7FA039B3D: __1cFSimbaEODBCKDescriptorOGetHeaderField6kMhpvpi_v_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FA11F45D: __1cFSimbaEODBCOStatementStateXDoColAttributeOnlyCount6MHphpl_nDstdEpair4Cp2Ch___ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FA12962D: __1cFSimbaEODBCWStatementStateExecutedQSQdDLColAttributeW6MHHpvhphpl_nDstdEpair4Cpn0BOStatementState_Ch___ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FA0F6AAE: __1cFSimbaEODBCJStatementQSQdDLColAttributeW6MHHpvhphpl_h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9EA3CC2: __1cFSimbaEODBCTSQdDLColAttributeTask4B_PDoSynchronously6Frn0BJStatement_rkn0COTaskParameters__h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9F10151: __1cGDoTask4nFSimbaEODBCTSQdDLColAttributeTask4B___6Fpkcpvrn7TAOTaskParameters__h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9EB815B: SQLColAttributeW (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FE23DC5E: SQLColAttributes_Internal (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE23E495: SQLColAttributes (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0xA69EE8: __1cFSimbaIODBCTestDCliQSqlColAttributes6MpvHH3hphpl_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE4CE7: __1cFSimbaIODBCTestJStatementQSqlColAttributes6MHHpvhphplrkn0BHOutcome_pkci_h_ (in /export/home/bamboo/Touchstone)
==900==  Address 0x37fedf48 is on thread 1's stack
==900==  664 bytes below stack pointer
==900==
==900== Invalid read of size 8
==900==    at 0x7FE3BD7D2: _Unw_jmp (in /lib/amd64/libc.so.1)
==900==    by 0x7FA12962D: __1cFSimbaEODBCWStatementStateExecutedQSQdDLColAttributeW6MHHpvhphpl_nDstdEpair4Cpn0BOStatementState_Ch___ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FA0F6AAE: __1cFSimbaEODBCJStatementQSQdDLColAttributeW6MHHpvhphpl_h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9EA3CC2: __1cFSimbaEODBCTSQdDLColAttributeTask4B_PDoSynchronously6Frn0BJStatement_rkn0COTaskParameters__h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9F10151: __1cGDoTask4nFSimbaEODBCTSQdDLColAttributeTask4B___6Fpkcpvrn7TAOTaskParameters__h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9EB815B: SQLColAttributeW (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FE23DC5E: SQLColAttributes_Internal (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE23E495: SQLColAttributes (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0xA69EE8: __1cFSimbaIODBCTestDCliQSqlColAttributes6MpvHH3hphpl_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE4CE7: __1cFSimbaIODBCTestJStatementQSqlColAttributes6MHHpvhphplrkn0BHOutcome_pkci_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD22371: __1c9uX__unnamed_Aj63VIoYhXFiiQColAttributeBasebEVerifyColAttributesNumberField6Mhl_v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD20720: __1c9uX__unnamed_Aj63VIoYhXFiiWSQdDLCOLATTRIBUTES_ERRORLexecuteTest6M_b_ (in /export/home/bamboo/Touchstone)
==900==  Address 0x37fedf78 is on thread 1's stack
==900==  1128 bytes below stack pointer
==900==
==900== Invalid read of size 8
==900==    at 0x7FE3BD7D2: _Unw_jmp (in /lib/amd64/libc.so.1)
==900==    by 0x7FA0F6AAE: __1cFSimbaEODBCJStatementQSQdDLColAttributeW6MHHpvhphpl_h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9EA3CC2: __1cFSimbaEODBCTSQdDLColAttributeTask4B_PDoSynchronously6Frn0BJStatement_rkn0COTaskParameters__h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9F10151: __1cGDoTask4nFSimbaEODBCTSQdDLColAttributeTask4B___6Fpkcpvrn7TAOTaskParameters__h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9EB815B: SQLColAttributeW (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FE23DC5E: SQLColAttributes_Internal (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE23E495: SQLColAttributes (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0xA69EE8: __1cFSimbaIODBCTestDCliQSqlColAttributes6MpvHH3hphpl_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE4CE7: __1cFSimbaIODBCTestJStatementQSqlColAttributes6MHHpvhphplrkn0BHOutcome_pkci_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD22776: __1c9uX__unnamed_Aj63VIoYhXFiiQColAttributeBasebCVerifyColAttributesWithError6MhrknFSimbaIODBCTestMThrowOutcome__v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD208FC: __1c9uX__unnamed_Aj63VIoYhXFiiWSQdDLCOLATTRIBUTES_ERRORLexecuteTest6M_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0x12F954D: __1cFSimbaETestECaseHrunTest6MrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____n0CLTEST_STATUS__ (in /export/home/bamboo/Touchstone)
==900==  Address 0x37fee258 is on thread 1's stack
==900==  664 bytes below stack pointer
==900==
==900== Invalid read of size 8
==900==    at 0x7FE3BD7D2: _Unw_jmp (in /lib/amd64/libc.so.1)
==900==    by 0x7F9EA3CC2: __1cFSimbaEODBCTSQdDLColAttributeTask4B_PDoSynchronously6Frn0BJStatement_rkn0COTaskParameters__h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9F10151: __1cGDoTask4nFSimbaEODBCTSQdDLColAttributeTask4B___6Fpkcpvrn7TAOTaskParameters__h_ (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7F9EB815B: SQLColAttributeW (in /export/home/bamboo/sol-crash/libInternalTest_debug.so)
==900==    by 0x7FE23DC5E: SQLColAttributes_Internal (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0x7FE23E495: SQLColAttributes (in /export/home/bamboo/libiodbc-3.52.8/usr/local/lib/libiodbc.so.2)
==900==    by 0xA69EE8: __1cFSimbaIODBCTestDCliQSqlColAttributes6MpvHH3hphpl_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE4CE7: __1cFSimbaIODBCTestJStatementQSqlColAttributes6MHHpvhphplrkn0BHOutcome_pkci_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD22776: __1c9uX__unnamed_Aj63VIoYhXFiiQColAttributeBasebCVerifyColAttributesWithError6MhrknFSimbaIODBCTestMThrowOutcome__v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD208FC: __1c9uX__unnamed_Aj63VIoYhXFiiWSQdDLCOLATTRIBUTES_ERRORLexecuteTest6M_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0x12F954D: __1cFSimbaETestECaseHrunTest6MrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____n0CLTEST_STATUS__ (in /export/home/bamboo/Touchstone)
==900==    by 0x1300335: __1cFSimbaETestGEngineHrunTest6Mpn0BECase__v_ (in /export/home/bamboo/Touchstone)
==900==  Address 0x37fee288 is on thread 1's stack
==900==  776 bytes below stack pointer
==900==
==900== Use of uninitialised value of size 8
==900==    at 0xAE161A: __1cFSimbaIODBCTestGHandleMCheckOutcome6kMrkn0BHOutcome_hpkci_v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE4D01: __1cFSimbaIODBCTestJStatementQSqlColAttributes6MHHpvhphplrkn0BHOutcome_pkci_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD22776: __1c9uX__unnamed_Aj63VIoYhXFiiQColAttributeBasebCVerifyColAttributesWithError6MhrknFSimbaIODBCTestMThrowOutcome__v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD208FC: __1c9uX__unnamed_Aj63VIoYhXFiiWSQdDLCOLATTRIBUTES_ERRORLexecuteTest6M_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0x12F954D: __1cFSimbaETestECaseHrunTest6MrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____n0CLTEST_STATUS__ (in /export/home/bamboo/Touchstone)
==900==    by 0x1300335: __1cFSimbaETestGEngineHrunTest6Mpn0BECase__v_ (in /export/home/bamboo/Touchstone)
==900==    by 0x12FFD8C: __1cFSimbaETestGEngineIRunTests6Mpn0BPTestEnvironment_i_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0xA66238: main (in /export/home/bamboo/Touchstone)
==900==
==900== Use of uninitialised value of size 8
==900==    at 0xAE1634: __1cFSimbaIODBCTestGHandleMCheckOutcome6kMrkn0BHOutcome_hpkci_v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE4D01: __1cFSimbaIODBCTestJStatementQSqlColAttributes6MHHpvhphplrkn0BHOutcome_pkci_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD22776: __1c9uX__unnamed_Aj63VIoYhXFiiQColAttributeBasebCVerifyColAttributesWithError6MhrknFSimbaIODBCTestMThrowOutcome__v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD208FC: __1c9uX__unnamed_Aj63VIoYhXFiiWSQdDLCOLATTRIBUTES_ERRORLexecuteTest6M_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0x12F954D: __1cFSimbaETestECaseHrunTest6MrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____n0CLTEST_STATUS__ (in /export/home/bamboo/Touchstone)
==900==    by 0x1300335: __1cFSimbaETestGEngineHrunTest6Mpn0BECase__v_ (in /export/home/bamboo/Touchstone)
==900==    by 0x12FFD8C: __1cFSimbaETestGEngineIRunTests6Mpn0BPTestEnvironment_i_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0xA66238: main (in /export/home/bamboo/Touchstone)
==900==
==900== Use of uninitialised value of size 8
==900==    at 0xAE1224: __1cFSimbaIODBCTestGHandleDlog6kM_pn0AETestNVerboseLogger__ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE1827: __1cFSimbaIODBCTestGHandleMCheckOutcome6kMrkn0BHOutcome_hpkci_v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE4D01: __1cFSimbaIODBCTestJStatementQSqlColAttributes6MHHpvhphplrkn0BHOutcome_pkci_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD22776: __1c9uX__unnamed_Aj63VIoYhXFiiQColAttributeBasebCVerifyColAttributesWithError6MhrknFSimbaIODBCTestMThrowOutcome__v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD208FC: __1c9uX__unnamed_Aj63VIoYhXFiiWSQdDLCOLATTRIBUTES_ERRORLexecuteTest6M_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0x12F954D: __1cFSimbaETestECaseHrunTest6MrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____n0CLTEST_STATUS__ (in /export/home/bamboo/Touchstone)
==900==    by 0x1300335: __1cFSimbaETestGEngineHrunTest6Mpn0BECase__v_ (in /export/home/bamboo/Touchstone)
==900==    by 0x12FFD8C: __1cFSimbaETestGEngineIRunTests6Mpn0BPTestEnvironment_i_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0xA66238: main (in /export/home/bamboo/Touchstone)
==900==
==900== Invalid read of size 1
==900==    at 0x7FE3AD4FD: mutex_lock_impl (in /lib/amd64/libc.so.1)
==900==    by 0x7FE3AD793: mutex_lock (in /lib/amd64/libc.so.1)
==900==    by 0x1317140: __1cFSimbaETestNVerboseLoggerDLog6MrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE184A: __1cFSimbaIODBCTestGHandleMCheckOutcome6kMrkn0BHOutcome_hpkci_v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE4D01: __1cFSimbaIODBCTestJStatementQSqlColAttributes6MHHpvhphplrkn0BHOutcome_pkci_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD22776: __1c9uX__unnamed_Aj63VIoYhXFiiQColAttributeBasebCVerifyColAttributesWithError6MhrknFSimbaIODBCTestMThrowOutcome__v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD208FC: __1c9uX__unnamed_Aj63VIoYhXFiiWSQdDLCOLATTRIBUTES_ERRORLexecuteTest6M_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0x12F954D: __1cFSimbaETestECaseHrunTest6MrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____n0CLTEST_STATUS__ (in /export/home/bamboo/Touchstone)
==900==    by 0x1300335: __1cFSimbaETestGEngineHrunTest6Mpn0BECase__v_ (in /export/home/bamboo/Touchstone)
==900==    by 0x12FFD8C: __1cFSimbaETestGEngineIRunTests6Mpn0BPTestEnvironment_i_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0xA66238: main (in /export/home/bamboo/Touchstone)
==900==  Address 0x650000017b is not stack'd, malloc'd or (recently) free'd
==900==
==900==
==900== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==900==  Access not within mapped region at address 0x650000017B
==900==    at 0x7FE3AD4FD: mutex_lock_impl (in /lib/amd64/libc.so.1)
==900==    by 0x7FE3AD793: mutex_lock (in /lib/amd64/libc.so.1)
==900==    by 0x1317140: __1cFSimbaETestNVerboseLoggerDLog6MrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE184A: __1cFSimbaIODBCTestGHandleMCheckOutcome6kMrkn0BHOutcome_hpkci_v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xAE4D01: __1cFSimbaIODBCTestJStatementQSqlColAttributes6MHHpvhphplrkn0BHOutcome_pkci_h_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD22776: __1c9uX__unnamed_Aj63VIoYhXFiiQColAttributeBasebCVerifyColAttributesWithError6MhrknFSimbaIODBCTestMThrowOutcome__v_ (in /export/home/bamboo/Touchstone)
==900==    by 0xD208FC: __1c9uX__unnamed_Aj63VIoYhXFiiWSQdDLCOLATTRIBUTES_ERRORLexecuteTest6M_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0x12F954D: __1cFSimbaETestECaseHrunTest6MrknDstdMbasic_string4Ccn0DLchar_traits4Cc__n0DJallocator4Cc_____n0CLTEST_STATUS__ (in /export/home/bamboo/Touchstone)
==900==    by 0x1300335: __1cFSimbaETestGEngineHrunTest6Mpn0BECase__v_ (in /export/home/bamboo/Touchstone)
==900==    by 0x12FFD8C: __1cFSimbaETestGEngineIRunTests6Mpn0BPTestEnvironment_i_b_ (in /export/home/bamboo/Touchstone)
==900==    by 0xA66238: main (in /export/home/bamboo/Touchstone)
==900==  If you believe this happened as a result of a stack
==900==  overflow in your program's main thread (unlikely but
==900==  possible), you can try to increase the size of the
==900==  main thread stack using the --main-stacksize= flag.
==900==  The main thread stack size used in this run was 8388608.
==900==
==900== HEAP SUMMARY:
==900==     in use at exit: 19,994,466 bytes in 157,661 blocks
==900==   total heap usage: 416,854 allocs, 259,193 frees, 56,010,824 bytes allocated
==900==
==900== LEAK SUMMARY:
==900==    definitely lost: 0 bytes in 0 blocks
==900==    indirectly lost: 0 bytes in 0 blocks
==900==      possibly lost: 10,903,944 bytes in 79,967 blocks
==900==    still reachable: 9,090,522 bytes in 77,694 blocks
==900==         suppressed: 0 bytes in 0 blocks
==900== Rerun with --leak-check=full to see details of leaked memory
==900==
==900== For counts of detected and suppressed errors, rerun with: -v
==900== Use --track-origins=yes to see where uninitialised values come from
==900== ERROR SUMMARY: 14 errors from 14 contexts (suppressed: 0 from 0)
Segmentation Fault

searching for _Uwn_jmp gives me this unfortunately (look at the name), which means that I was seeing the same thing for what I dismissed as gcc/cc incompatibility.

The same tests work fine for 32-bit.

Any ideas?

Is there any way you can post a [mcve]? `r14` is a call-preserved register, and not an arg-passing register, in the [SysV AMD64 ABI](http://www.x86-64.org/documentation.html). Functions that want to use it will push it in their prologue and pop it in their epilogue. But they don't care what their caller was using it for. I haven't looked at the asm for exception-catching code. IDK what they're supposed to be able to assume about register state. — Peter Cordes, Jul 15 '16 at 01:02
Does the problem only happen under Valgrind? I've noted also that you used Solaris Studio 12.4 to compile and later that you're using LD_PRELOAD to load the GNU C++ run-time library. If you compiled your code with Solaris Studio and didn't use the GNU C++ run-time, that mixing of multiple C++ run-time libraries likely will not end well. — Andrew Henle, Jul 15 '16 at 08:54
This GCC bug looks relevant: [Bug 59788 - Mixing libc and libgcc_s unwinders on 64-bit Solaris 10+/x86 breaks EH](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59788) — Andrew Henle, Jul 15 '16 at 11:35
And an interesting thread on mixing C++ run-times: http://gcc-help.gcc.gnu.narkive.com/1ab74CKr/loading-multiple-c-runtimes-but-not-mixing-abis — Andrew Henle, Jul 15 '16 at 11:56
@AndrewHenle - No, it happens with or without valgrind. The LD_PRELOAD thing was more voodoo debugging, since nothing involved was compiled with gcc (and ldd doesn't show any dependency on libgcc_s), but it seemed similar to that bug. It crashes with or without the use of LD_PRELOAD, that seems to have no effect. — Bwmat, Jul 15 '16 at 16:22
So you're running into an issue that has the same symptoms as a library-ordering issue with `libgcc_s` on 64-bit Solaris and just like that issue the 32-bit version works, but you didn't compile anything with GCC and have no dependencies on `libgcc_s`? So it looks like a duck, quacks like a duck, but it's an octopus? Ouch. OK, then, what does `ldd -s -r` show for the program's dynamic dependencies? Both 32- and 64-bit, asthe 32-bit one works and any differences could be important. Using `-s` will show the search path, and `-r` will force all symbol lookups to be done - those might be useful. — Andrew Henle, Jul 15 '16 at 18:46

Peter Cordes · Accepted Answer · 2016-07-15T20:53:55.040

(not a full answer, but this seemed too long for comments).

TL:DR: check the ABI instead of guessing about which part is wrong.

r14 is a call-preserved register (and not used for arg-passing) in the SysV AMD64 ABI. Functions that want to use it will push it in their prologue and pop it in their epilogue. But they don't care what their caller was using it for. They just have to make sure it has the same value on return as it did on entry.

The register state on entry to a catch clause seems to be documented in Section 6.2.6 of the ABI:

Transferring Control to a Landing Pad

...

Prior to executing code in the landing pad, the unwind library restores registers not altered by the personality routine, using the context record, to their state in that frame before the call that threw the exception, as follows. All registers specified as callee-saved by the base ABI are restored, [and %rsi, %rdi, %rdx, %rcx are used to pass info about the exception].

The original end of that last sentence in the ABI is confusingly worded. My paraphrase inside [ ] may be wrong. (See the last paragraph of that subsection).

Based on this and previous sections (describing how cleanup code for each frame is called even if it doesn't have a catch), I'm almost certain that call-preserved registers are restored in a catch. This is the same mechanism that runs destructors during unwinding, which is necessary.

So it sounds like the error is that r14 isn't being restored correctly. The function that catches the exception and tries to use r14 isn't doing anything wrong.

Maybe try including destructors that print some logging info in some of the functions that will have their stack frame unwound? Maybe have them check a global to decide whether to print, so you can set a global and then make the call that will result in an exception being thrown, so you don't get a screen full of log messages from calls that don't throw.

Maybe destructors/cleanup aren't being called correctly during phase2 of unwinding, if you didn't see r14 being restored.

... put a hardware write watch on the stack where the register was getting spilled to, but it doesn't get hit until after returning into touchstone

You should have been using a read watchpoint to look for it being read during the unwind cleanup code for the function that spilled it. It's not surprising that nothing overwrites the spill location until after entering the catch clause.

I'm not too familiar with this type of debugging, thanks for the link to the abi, I'll take a look at it tomorrow — Bwmat, Jul 15 '16 at 06:51
None of the code between the throw and catch is asm, just plain old c++ — Bwmat, Jul 15 '16 at 06:53
@Bwmat: See also the [x86 tag wiki](http://stackoverflow.com/tags/x86/info) for lots of helpful links about x86 asm. — Peter Cordes, Jul 15 '16 at 07:03
I read section 6.2 about exception handling in the abi spec you linked, it doesn't really explicitly say anything about register preservation or anything, but I can only assume that call-preserved registers are supposed to be restored... — Bwmat, Jul 15 '16 at 18:00
@Bwmat: I had a look. The key info is buried in with a bunch of stuff that is probably working correctly. I made an edit based on what I found. It sounds like you should probably report this as a bug to the compiler vendor. — Peter Cordes, Jul 15 '16 at 20:55

Crash caused by invalid exception unwinding?

1 Answers1

Linked