Segfault in TSAN in Boost CI - Need the help of an expert

Question

I am debugging A segfault reported by TSAN in the CI of Boost.Beast.

I strongly believe it to be a false positive, but I don't know what to look for in order to suppress it.

It seems to me from the stack trace that the code is correctly instrumented. The code passes all other tests, inclding valgrind, ubsan, etc.

I'm hoping some kind expert may put me out of my misery.

Here is the output:

====== BEGIN OUTPUT ======
beast.http.read
ThreadSanitizer:DEADLYSIGNAL
==132842==ERROR: ThreadSanitizer: SEGV on unknown address 0x7ff5d9cff000 (pc 0x7ff5dceba0d0 bp 0x000000000000 sp 0x7ff5d9c3d910 T132844)
==132842==The signal is caused by a READ memory access.
    #0 __sanitizer::StackDepotBase<__sanitizer::StackDepotNode, 1, 20>::Put(__sanitizer::StackTrace, bool*) <null> (libtsan.so.2+0xba0d0)
    #1 __tsan::CurrentStackId(__tsan::ThreadState*, unsigned long) <null> (libtsan.so.2+0x8c48f)
    #2 __sanitizer::DD::MutexInit(__sanitizer::DDCallback*, __sanitizer::DDMutex*) <null> (libtsan.so.2+0xac534)
    #3 __tsan::DDMutexInit(__tsan::ThreadState*, unsigned long, __tsan::SyncVar*) <null> (libtsan.so.2+0x9a3f8)
    #4 __tsan::MetaMap::GetSync(__tsan::ThreadState*, unsigned long, unsigned long, bool, bool) <null> (libtsan.so.2+0xa85dc)
    #5 __tsan_atomic32_fetch_add <null> (libtsan.so.2+0x783e9)
    #6 __gnu_cxx::__exchange_and_add(int volatile*, int) /usr/include/c++/12/ext/atomicity.h:66 (read+0x4188f8)
    #7 __gnu_cxx::__exchange_and_add_dispatch(int*, int) /usr/include/c++/12/ext/atomicity.h:101 (read+0x4188f8)
    #8 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use() /usr/include/c++/12/bits/shared_ptr_base.h:187 (read+0x4188f8)
    #9 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /usr/include/c++/12/bits/shared_ptr_base.h:361 (read+0x40c592)
    #10 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/include/c++/12/bits/shared_ptr_base.h:1071 (read+0x418fee)
    #11 std::__shared_ptr<boost::asio::detail::strand_executor_service::strand_impl, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/include/c++/12/bits/shared_ptr_base.h:1524 (read+0x420f83)
    #12 std::shared_ptr<boost::asio::detail::strand_executor_service::strand_impl>::~shared_ptr() /usr/include/c++/12/bits/shared_ptr.h:175 (read+0x420faf)
    #13 boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>::~invoker() <null> (read+0x471f5f)
    #14 boost::asio::detail::executor_op<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>, boost::asio::detail::recycling_allocator<void, boost::asio::detail::thread_info_base::default_tag>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) <null> (read+0x48b0ac)
    #15 boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long) boost/asio/detail/scheduler_operation.hpp:40 (read+0x4fa38e)
    #16 boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) boost/asio/detail/impl/scheduler.ipp:492 (read+0x4e8835)
    #17 boost::asio::detail::scheduler::run(boost::system::error_code&) boost/asio/detail/impl/scheduler.ipp:210 (read+0x4e74fb)
    #18 boost::asio::io_context::run() boost/asio/impl/io_context.ipp:63 (read+0x4dc122)
    #19 boost::beast::test::enable_yield_to::enable_yield_to(unsigned long)::{lambda()#1}::operator()() const <null> (read+0x412363)
    #20 void std::__invoke_impl<void, boost::beast::test::enable_yield_to::enable_yield_to(unsigned long)::{lambda()#1}>(std::__invoke_other, boost::beast::test::enable_yield_to::enable_yield_to(unsigned long)::{lambda()#1}&&) <null> (read+0x4a1f92)
    #21 std::__invoke_result<boost::beast::test::enable_yield_to::enable_yield_to(unsigned long)::{lambda()#1}>::type std::__invoke<boost::beast::test::enable_yield_to::enable_yield_to(unsigned long)::{lambda()#1}>(boost::beast::test::enable_yield_to::enable_yield_to(unsigned long)::{lambda()#1}&&) <null> (read+0x49f9dc)
    #22 void std::thread::_Invoker<std::tuple<boost::beast::test::enable_yield_to::enable_yield_to(unsigned long)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) <null> (read+0x49c90a)
    #23 std::thread::_Invoker<std::tuple<boost::beast::test::enable_yield_to::enable_yield_to(unsigned long)::{lambda()#1}> >::operator()() <null> (read+0x49941e)
    #24 std::thread::_State_impl<std::thread::_Invoker<std::tuple<boost::beast::test::enable_yield_to::enable_yield_to(unsigned long)::{lambda()#1}> > >::_M_run() <null> (read+0x494c2a)
    #25 execute_native_thread_routine <null> (libstdc++.so.6+0xdbb72)
    #26 __tsan_thread_start_func <null> (libtsan.so.2+0x393ef)
    #27 start_thread <null> (libc.so.6+0x8ce2c)
    #28 clone3 <null> (libc.so.6+0x1121af)

ThreadSanitizer can not provide additional info.
SUMMARY: ThreadSanitizer: SEGV (/lib64/libtsan.so.2+0xba0d0) in __sanitizer::StackDepotBase<__sanitizer::StackDepotNode, 1, 20>::Put(__sanitizer::StackTrace, bool*)
==132842==ABORTING

The code being tested is latest master branch.

Command line to reproduce:

$ ./b2 toolset=gcc thread-sanitizer=norecover link=static variant=debug libs/beast/test -q -d+2 -j1

My compiler info:

$ gcc --version
gcc (GCC) 12.1.1 20220507 (Red Hat 12.1.1-1)
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

My OS is Fedora 36. But we see this happen on Ubuntu as well.

What code can be used to reproduce it? What version of compiler? FWIW some other Boost libraries have _long-standing_ issues with TSAN (https://github.com/boostorg/lockfree/issues/22 e.g. from code like here https://stackoverflow.com/q/72130807/85371) — sehe, Aug 15 '22 at 09:21
@RichardHodges I have exactly the same setup as you. I'll build boost 1.80.0 and see if it also has the same issue. — Ted Lyngmo, Aug 15 '22 at 09:37
@RichardHodges Sorry to say, the 1.80.0 release has the same issue :-( — Ted Lyngmo, Aug 15 '22 at 09:53
Yes, 1.80 is only a few commits behind master. I strongly believe it's a false positive, but I have no idea what to do about it. TSAN is a Dark Art to me. Any Ideas? — Richard Hodges, Aug 15 '22 at 10:00
Hmm, no. Perhaps it'd be best to ask at the boost developers mailing list (signup [here](https://lists.boost.org/mailman/listinfo.cgi/boost)) to see if anyone there has an idea. — Ted Lyngmo, Aug 15 '22 at 10:11
Already done - I'm in full distress call mode https://lists.boost.org/Archives/boost/2022/08/253358.php — Richard Hodges, Aug 15 '22 at 10:25

sehe · Answer 1 · 2022-08-15T18:02:07.073

Using the b2 line I could repro the SEGV on linux using master (00293a6adb5 from the superproject).

I started with a - to me - more convenient setup based on CMake. I modified the CMake to use thread,undefined sanitizers instead of address,undefined for VARIANT=ubasan.

Interestingly, it doesn't segfault. It does however seem to have a legit TSAN violation in basic_stream.cpp, where the effective flags are:

{
    "directory": "/backup/cloudbackup/custom_ex/superboost/libs/beast/build/test/beast/core",
    "command": "/home/sehe/bin/g++-10 -DBOOST_ALL_STATIC_LINK=1 -DBOOST_ASIO_DISABLE_BOOST_ARRAY=1 -DBOOST_ASIO_DISABLE_BOOST_BIND=1 -DBOOST_ASIO_DISABLE_BOOST_DATE_TIME=1 -DBOOST_ASIO_DISABLE_BOOST_REGEX=1 -DBOOST_ASIO_NO_DEPRECATED=1 -DBOOST_ASIO_SEPARATE_COMPILATION=1 -DBOOST_BEAST_ALLOW_DEPRECATED -DBOOST_BEAST_SEPARATE_COMPILATION=1 -DBOOST_BEAST_TESTS -DBOOST_COROUTINES_NO_DEPRECATION_WARNING=1 -I/backup/cloudbackup/custom_ex/superboost/libs/beast/include -I/backup/cloudbackup/custom_ex/superboost/libs/beast/. -I/backup/cloudbackup/custom_ex/superboost/libs/beast/test/./extern -I/backup/cloudbackup/custom_ex/superboost/libs/beast/test/./extras/include -isystem /backup/cloudbackup/custom_ex/superboost  -std=c++11 -Wall -Wextra -Wpedantic -Wno-unused-parameter -DBOOST_BEAST_NO_SLOW_TESTS=1 -msse4.2 -funsigned-char -fno-omit-frame-pointer -fsanitize=thread,undefined -O2 -g -DNDEBUG -pthread -o CMakeFiles/tests-beast-core.dir/basic_stream.cpp.o -c /backup/cloudbackup/custom_ex/superboost/libs/beast/test/beast/core/basic_stream.cpp",
    "file": "/backup/cloudbackup/custom_ex/superboost/libs/beast/test/beast/core/basic_stream.cpp"
},

Breaking it down for readability:

g++-10
-DBOOST_ALL_STATIC_LINK=1
-DBOOST_ASIO_DISABLE_BOOST_ARRAY=1
-DBOOST_ASIO_DISABLE_BOOST_BIND=1
-DBOOST_ASIO_DISABLE_BOOST_DATE_TIME=1
-DBOOST_ASIO_DISABLE_BOOST_REGEX=1
-DBOOST_ASIO_NO_DEPRECATED=1
-DBOOST_ASIO_SEPARATE_COMPILATION=1
-DBOOST_BEAST_ALLOW_DEPRECATED
-DBOOST_BEAST_SEPARATE_COMPILATION=1
-DBOOST_BEAST_TESTS
-DBOOST_COROUTINES_NO_DEPRECATION_WARNING=1
-I/superboost/libs/beast/include
-I/superboost/libs/beast/.
-I/superboost/libs/beast/test/./extern
-I/superboost/libs/beast/test/./extras/include
-isystem /superboost
-std=c++11
-Wall
-Wextra
-Wpedantic
-Wno-unused-parameter
-DBOOST_BEAST_NO_SLOW_TESTS=1
-msse4.2
-funsigned-char
-fno-omit-frame-pointer
-fsanitize=thread,undefined
-O2 -g
-DNDEBUG
-pthread
-o CMakeFiles/tests-beast-core.dir/basic_stream.cpp.o
-c /superboost/libs/beast/test/beast/core/basic_stream.cpp

The reported diagnostic: https://paste.ubuntu.com/p/6SKjmZ9wFT/ (lines truncated for SO):

beast.core.basic_stream
==================
WARNING: ThreadSanitizer: data race (pid=24051)
  Write of size 8 at 0x7b0400000030 by thread T1:
    #0 pipe ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1726 (libtsan.so.0+0x3e574)
    #1 __sanitizer::IsAccessibleMemoryRange(unsigned long, unsigned long) ../../../../src/libsanitizer/sanitizer_common/sanitizer_posix_libcdep.cpp:281 (libu...
    #2 operator() /superboost/libs/beast/test/beast/core/basic_stream.cpp:228 (tests-beast-core+0x4f0271)
    #3 operator() /superboost/boost/asio/detail/bind_handler.hpp:171 (tests-beast-core+0x4f0271)
    #4 invoke<boost::asio::detail::binder1<boost::beast::(anonymous namespace)::test_server::test_server(boost::beast::string_view, boost::asio::ip::tcp::end...
    #5 complete<boost::asio::detail::binder1<boost::beast::(anonymous namespace)::test_server::test_server(boost::beast::string_view, boost::asio::ip::tcp::e...
    #6 do_complete /superboost/boost/asio/detail/reactive_socket_accept_op.hpp:150 (tests-beast-core+0x4f0271)
    #7 boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long) /superboost/boost/asio/detail/scheduler_ope...
    #8 boost::asio::detail::epoll_reactor::descriptor_state::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, ...
    #9 boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long) /superboost/boost/asio/detail/scheduler_ope...
    #10 boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info...
    #11 boost::asio::detail::scheduler::run(boost::system::error_code&) /superboost/boost/asio/detail/impl/scheduler.ipp:210 (tests-beast-core+0x90d424)
    #12 boost::asio::io_context::run() /superboost/boost/asio/impl/io_context.ipp:63 (tests-beast-core+0x91bf39)
    #13 operator() /superboost/libs/beast/test/beast/core/basic_stream.cpp:234 (tests-beast-core+0x4cf1f9)
    #14 __invoke_impl<void, boost::beast::(anonymous namespace)::test_server::test_server(boost::beast::string_view, boost::asio::ip::tcp::endpoint, std::ost...
    #15 __invoke<boost::beast::(anonymous namespace)::test_server::test_server(boost::beast::string_view, boost::asio::ip::tcp::endpoint, std::ostream&)::<la...
    #16 _M_invoke<0> /usr/include/c++/10/thread:264 (tests-beast-core+0x4cf1f9)
    #17 operator() /usr/include/c++/10/thread:271 (tests-beast-core+0x4cf1f9)
    #18 _M_run /usr/include/c++/10/thread:215 (tests-beast-core+0x4cf1f9)
    #19 execute_native_thread_routine ../../../../../src/libstdc++-v3/src/c++11/thread.cc:82 (libstdc++.so.6+0xd44bf)

  Previous write of size 8 at 0x7b0400000030 by main thread:
    #0 pipe ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1726 (libtsan.so.0+0x3e574)
    #1 __sanitizer::IsAccessibleMemoryRange(unsigned long, unsigned long) ../../../../src/libsanitizer/sanitizer_common/sanitizer_posix_libcdep.cpp:281 (libu...
    #2 void boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> >::initiate_async_...
    #3 void boost::asio::detail::completion_handler_async_result<boost::beast::basic_stream<boost::asio::ip::tcp, boost::asio::io_context::basic_executor_typ...
    #4 boost::asio::constraint<boost::asio::detail::async_result_has_initiate_memfn<boost::beast::basic_stream<boost::asio::ip::tcp, boost::asio::io_context:...
    #5 decltype ((async_initiate<boost::beast::basic_stream<boost::asio::ip::tcp, boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>, bo...
    #6 boost::beast::basic_stream<boost::asio::ip::tcp, boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>, boost::beast::unlimited_rate...
    #7 boost::beast::basic_stream<boost::asio::ip::tcp, boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>, boost::beast::unlimited_rate...
    #8 boost::beast::basic_stream<boost::asio::ip::tcp, boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>, boost::beast::unlimited_rate...
    #9 void boost::beast::basic_stream<boost::asio::ip::tcp, boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>, boost::beast::unlimited...
    #10 void boost::asio::detail::completion_handler_async_result<boost::beast::basic_stream_test::handler, void (boost::system::error_code, unsigned long)>:...
    #11 boost::asio::constraint<boost::asio::detail::async_result_has_initiate_memfn<boost::beast::basic_stream_test::handler, void (boost::system::error_cod...
    #12 boost::asio::async_result<std::decay<boost::beast::basic_stream_test::handler>::type, void (boost::system::error_code, unsigned long)>::return_type b...
    #13 boost::beast::basic_stream_test::testRead() /superboost/libs/beast/test/beast/core/basic_stream.cpp:488 (tests-beast-core+0x678b01)
    #14 boost::beast::basic_stream_test::run() /superboost/libs/beast/test/beast/core/basic_stream.cpp:1401 (tests-beast-core+0x70b193)
    #15 void boost::beast::unit_test::suite::run<void>(boost::beast::unit_test::runner&) /superboost/libs/beast/include/boost/beast/_experimental/unit_test/s...
    #16 void boost::beast::unit_test::suite::operator()<void>(boost::beast::unit_test::runner&) /superboost/libs/beast/include/boost/beast/_experimental/unit...
    #17 boost::beast::unit_test::make_suite_info<boost::beast::basic_stream_test>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<cha...
    #18 void std::__invoke_impl<void, boost::beast::unit_test::make_suite_info<boost::beast::basic_stream_test>(std::__cxx11::basic_string<char, std::char_tr...
    #19 std::enable_if<std::__and_<std::is_void<void>, std::__is_invocable<boost::beast::unit_test::make_suite_info<boost::beast::basic_stream_test>(std::__c...
    #20 std::_Function_handler<void (boost::beast::unit_test::runner&), boost::beast::unit_test::make_suite_info<boost::beast::basic_stream_test>(std::__cxx1...
    #21 std::function<void (boost::beast::unit_test::runner&)>::operator()(boost::beast::unit_test::runner&) const /usr/include/c++/10/bits/std_function.h:62...
    #22 boost::beast::unit_test::suite_info::run(boost::beast::unit_test::runner&) const /superboost/libs/beast/include/boost/beast/_experimental/unit_test/s...
    #23 bool boost::beast::unit_test::runner::run<void>(boost::beast::unit_test::suite_info const&) /superboost/libs/beast/include/boost/beast/_experimental/...
    #24 bool boost::beast::unit_test::runner::run_each<boost::beast::unit_test::suite_list>(boost::beast::unit_test::suite_list const&) /superboost/libs/beas...
    #25 main /superboost/libs/beast/include/boost/beast/_experimental/unit_test/main.ipp:82 (tests-beast-core+0x4cc1d3)

  Thread T1 (tid=24054, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x5fe84)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) /build/gcc-11-YRKbe7/gcc-11-...
    #2 boost::beast::basic_stream_test::testRead() /superboost/libs/beast/test/beast/core/basic_stream.cpp:484 (tests-beast-core+0x67891b)
    #3 boost::beast::basic_stream_test::run() /superboost/libs/beast/test/beast/core/basic_stream.cpp:1401 (tests-beast-core+0x70b193)
    #4 void boost::beast::unit_test::suite::run<void>(boost::beast::unit_test::runner&) /superboost/libs/beast/include/boost/beast/_experimental/unit_test/su...
    #5 void boost::beast::unit_test::suite::operator()<void>(boost::beast::unit_test::runner&) /superboost/libs/beast/include/boost/beast/_experimental/unit_...
    #6 boost::beast::unit_test::make_suite_info<boost::beast::basic_stream_test>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char...
    #7 void std::__invoke_impl<void, boost::beast::unit_test::make_suite_info<boost::beast::basic_stream_test>(std::__cxx11::basic_string<char, std::char_tra...
    #8 std::enable_if<std::__and_<std::is_void<void>, std::__is_invocable<boost::beast::unit_test::make_suite_info<boost::beast::basic_stream_test>(std::__cx...
    #9 std::_Function_handler<void (boost::beast::unit_test::runner&), boost::beast::unit_test::make_suite_info<boost::beast::basic_stream_test>(std::__cxx11...
    #10 std::function<void (boost::beast::unit_test::runner&)>::operator()(boost::beast::unit_test::runner&) const /usr/include/c++/10/bits/std_function.h:62...
    #11 boost::beast::unit_test::suite_info::run(boost::beast::unit_test::runner&) const /superboost/libs/beast/include/boost/beast/_experimental/unit_test/s...
    #12 bool boost::beast::unit_test::runner::run<void>(boost::beast::unit_test::suite_info const&) /superboost/libs/beast/include/boost/beast/_experimental/...
    #13 bool boost::beast::unit_test::runner::run_each<boost::beast::unit_test::suite_list>(boost::beast::unit_test::suite_list const&) /superboost/libs/beas...
    #14 main /superboost/libs/beast/include/boost/beast/_experimental/unit_test/main.ipp:82 (tests-beast-core+0x4cc1d3)

SUMMARY: ThreadSanitizer: data race ../../../../src/libsanitizer/sanitizer_common/sanitizer_posix_libcdep.cpp:281 in __sanitizer::IsAccessibleMemoryRange(uns...
==================
559ms, 1 suite, 1 case, 217 tests total, 0 failures
ThreadSanitizer: reported 1 warnings

Given this observation, I thought to see whether excluding the offending TU (basic_stream.cpp) from the b2 build removes the SEGV. No such luck.

On the contrary, the SEGV manfifest with the following TUs:

module	TU
test/beast/core	`buffered_read_stream.cpp`
test/beast/http	`read.cpp`
test/beast/http	`write.cpp`
test/beast/websocket	`close.cpp`
test/beast/websocket	`handshake.cpp`
test/beast/websocket	`ping.cpp`
test/beast/websocket	`read2.cpp`
test/beast/websocket	`write.cpp`
test/doc	`http_examples.cpp`

Dropping these TUs from their respective test/**/Jamfile allows all remaining tests to pass TSAN under b2. Now, I did some soul searching and e.g. unique include diving using a script like:

#!/bin/bash -i
alias PP='find build/ -name *.cpp.i'
PP -delete

make -C build/test/beast/core buffered_read_stream.i
make -C build/test/beast/http read.i
make -C build/test/beast/http write.i
make -C build/test/beast/websocket close.i
make -C build/test/beast/websocket handshake.i
make -C build/test/beast/websocket ping.i
make -C build/test/beast/websocket read2.i
make -C build/test/beast/websocket write.i
make -C build/test/doc http_examples.i

PP | nl
n=$(PP | wc -l)
set -x

export LANG=C 
PP -exec sort -b {} \+ | uniq -dc | grep -wP "^\s*$n # 1" | grep -P '1( 3 4)?$' | nl

Which uncovers a common subset of includes of 854 includes. 40 of the beast headers are in that consistent set, but 208 are asio headers.

Questions from here:

why is SEGV not happening in the CMake build?
are there headers in the common subset that do not appear in the TUs that don't trip TSAN up?
is a recent change in Asio relevant?

Choosing to address these 3., 1., 2. (optimizing for return-on-effort)

3. Is a recent Asio change involved? [YES]

Doing the b2 test with only Asio reverted to 1.79.0 (e929e5cf Merge asio from 'develop') passes all the tests cleanly.

Just to check that no compiler flags were harmed in the process e.g. buffered_read_stream.cpp showed the same exact commands:

gcc.compile.c++ bin.v2/libs/beast/test/beast/core/buffered_read_stream.test/gcc-10.0/debug/link-static/thread-sanitizer-norecover/threading-multi/visibility-hidden/buffered_read_stream.o

    "g++-10"   -fvisibility-inlines-hidden -fsanitize=thread -fno-sanitize-recover=thread -fno-omit-frame-pointer -m64 -pthread -O0 -fno-inline -Wall -g -fvisibility=hidden  -DBOOST_ALL_NO_LIB=1 -DBOOST_ASIO_DISABLE_BOOST_ARRAY=1 -DBOOST_ASIO_DISABLE_BOOST_BIND=1 -DBOOST_ASIO_DISABLE_BOOST_DATE_TIME=1 -DBOOST_ASIO_DISABLE_BOOST_REGEX=1 -DBOOST_ASIO_NO_DEPRECATED=1 -DBOOST_ASIO_SEPARATE_COMPILATION -DBOOST_ATOMIC_STATIC_LINK=1 -DBOOST_BEAST_ALLOW_DEPRECATED -DBOOST_BEAST_SEPARATE_COMPILATION -DBOOST_BEAST_TESTS -DBOOST_COROUTINES_NO_DEPRECATION_WARNING=1 -DBOOST_FILESYSTEM_STATIC_LINK=1 -D_GNU_SOURCE=1 -D_XOPEN_SOURCE=600  -I"." -I"libs/beast" -I"libs/beast/test/extern" -I"libs/beast/test/extras/include"  -c -o "bin.v2/libs/beast/test/beast/core/buffered_read_stream.test/gcc-10.0/debug/link-static/thread-sanitizer-norecover/threading-multi/visibility-hidden/buffered_read_stream.o" "libs/beast/test/beast/core/buffered_read_stream.cpp"

gcc.link bin.v2/libs/beast/test/beast/core/buffered_read_stream.test/gcc-10.0/debug/link-static/thread-sanitizer-norecover/threading-multi/visibility-hidden/buffered_read_stream

    "g++-10"    -o "bin.v2/libs/beast/test/beast/core/buffered_read_stream.test/gcc-10.0/debug/link-static/thread-sanitizer-norecover/threading-multi/visibility-hidden/buffered_read_stream" -Wl,--start-group "bin.v2/libs/beast/test/beast/core/buffered_read_stream.test/gcc-10.0/debug/link-static/thread-sanitizer-norecover/threading-multi/visibility-hidden/buffered_read_stream.o" "bin.v2/libs/beast/test/gcc-10.0/debug/link-static/thread-sanitizer-norecover/threading-multi/visibility-hidden/lib-test.a" "bin.v2/libs/beast/gcc-10.0/debug/link-static/thread-sanitizer-norecover/threading-multi/visibility-hidden/lib-asio.a" "bin.v2/libs/beast/gcc-10.0/debug/link-static/thread-sanitizer-norecover/threading-multi/visibility-hidden/lib-beast.a" "bin.v2/libs/coroutine/build/gcc-10.0/debug/link-static/thread-sanitizer-norecover/threading-multi/visibility-hidden/libboost_coroutine.a" "bin.v2/libs/context/build/gcc-10.0/debug/link-static/thread-sanitizer-norecover/threading-multi/visibility-hidden/libboost_context.a" "bin.v2/libs/filesystem/build/gcc-10.0/debug/link-static/thread-sanitizer-norecover/threading-multi/visibility-hidden/libboost_filesystem.a" "bin.v2/libs/atomic/build/gcc-10.0/debug/link-static/thread-sanitizer-norecover/threading-multi/visibility-hidden/libboost_atomic.a"  -Wl,-Bstatic  -Wl,-Bdynamic -lrt -Wl,--end-group -m64 -pthread -g -fvisibility=hidden -fvisibility-inlines-hidden -fsanitize=thread -fno-sanitize-recover=thread -fno-omit-frame-pointer

So, now we know that something inside the 208 Asio headers must be involved.

1. Why is SEGV not happening in the CMake build?

Surely, here we should be able to spot difference in compilation flags? Ever-so-slightly redacted to remove spelling differences (left = CMake, right = b2):

I used the process of elimination, figured out that the culprit is -fno-inline -O0. Somehow it leads to recursive TSAN errors:

beast.core.buffered_read_stream
ThreadSanitizer:DEADLYSIGNAL
==2827==ERROR: ThreadSanitizer: SEGV on unknown address 0x60000212fff8 (pc 0x7f319a938bfc bp 0x7f3196fbe1c0 sp 0x7f3196fbe1a8 T2829)
==2827==The signal is caused by a WRITE memory access.
ThreadSanitizer:DEADLYSIGNAL
ThreadSanitizer: nested bug in the same thread, aborting.

Observations without -fno-inline:

from -O2 the symptoms go away
at -O1 the symptoms go away if NDEBUG is defined
at -O0 the symptom is there regardless

This suggests that a NDEBUG-sensitive piece of code could be involved. This might be a lead to guide minimization.

Comparing preprocessed sources may highlight specific possible causes. My main suspicions are the revamped spawned_thread_base in asio/spawn.hpp, source-locations and or cancellation slots.

As a courtesy, here's the preprocessed-buffered-stream-reader.zip containing 4 files (/home/sehe/{with,without}-NDEBUG.i.asio1.{79,80}.0).

I ran out of steam (long after I should have run out of time) so I left question 2. ("are there headers in the common subset that do not appear in the TUs that don't trip TSAN up") unanswered. To be honest, given the leads that came out of the questions 3. and 1. I would not thing question 2. the most effective way to generate leads at this point — sehe, Aug 15 '22 at 17:53

score 3 · Answer 2 · answered Aug 16 '22 at 13:47

Thanks everyone who chipped in on this. I was was able to get some help from the legendary Chris Kohlhoff, author of the Asio library.

Quoting:

It seems that thread-sanitizer does not correctly handle coroutine/fiber stacks that migrate between threads. This would either be considered a bug in thread-sanitizer, or perhaps a feature request for it.

The issue is in the thread sanitiser library itself.

The workaround in this case is to restructure the code so that the initiation of the fiber takes place on the same thread as the one in which it makes progress.

Here's the old (correct but not TSAN-compatible) code:

template<class F0, class... FN>
inline
void
enable_yield_to::
spawn(F0&& f, FN&&... fn)
{
    asio::spawn(ioc_,
        [&](yield_context yield)
        {
            f(yield);
            std::lock_guard<std::mutex> lock{m_};
            if(--running_ == 0)
                cv_.notify_all();
        }
        , boost::coroutines::attributes(2 * 1024 * 1024));
    spawn(fn...);
}

And here's the code with the workaround:

template<class F0, class... FN>
inline
void
enable_yield_to::
spawn(F0&& f, FN&&... fn)
{
    // dispatch of spawn is a workaround for
    // https://github.com/boostorg/beast/issues/2499
    asio::dispatch(ioc_, 
        [&]
        {
            asio::spawn(ioc_,
                [&](yield_context yield)
                {
                    f(yield);
                    std::lock_guard<std::mutex> lock{m_};
                    if(--running_ == 0)
                        cv_.notify_all();
                }
                , boost::coroutines::attributes(2 * 1024 * 1024));
        });
    spawn(fn...);
}

That's pretty awesome. The new spawn implementation (which rocks btw) was always the suspect, but I'm impressed with the speed and brevity of the workaround. Is anything known about its inclusion and any performance ramifications? I imagine most code using `spawn` does so relatively infrequently — sehe, Aug 16 '22 at 13:51
I just realized that there are compilation defines in boost to enable ASAN support, perhaps one exists/can be added for TSAN support. Though it feels iffy to make a strategic difference like this depend on a compilation flag, as it can lead to situations where TSAN builds test something different than the released program. — sehe, Aug 16 '22 at 14:40
@RichardHodges Just noticed your mail on the boost mailing list and had to pop over here. Great news! — Ted Lyngmo, Aug 16 '22 at 21:32

Segfault in TSAN in Boost CI - Need the help of an expert

2 Answers2

3. Is a recent Asio change involved? [YES]

1. Why is SEGV not happening in the CMake build?