TSAN Failures

[mcdougallryan]

What has been your experience with TSAN? On a simple GTEST we get a lot off output, but inspecting the source it's not obvious that this is a problem. Reproduced below is one of the more common errors. ASAN passes, and this should be standard C++ passing arguments by stack (not shared heap variables that need protecting). Is this a known issue? Do you have existing suppressions for false positives?

WARNING: ThreadSanitizer: data race (pid=50845)
  Write of size 1 at 0x7b6000000804 by thread T11:
    #0 getpwuid ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:1929 (libtsan.so.0+0x36125)
    #1 iox::posix::PosixUser::getUserName(unsigned int) <null> (iceoryx_integration_test+0x25d708)
    #2 iox::posix::PosixUser::PosixUser(unsigned int) <null> (iceoryx_integration_test+0x25e8e6)
    #3 iox::roudi::RouDi::processMessage(iox::runtime::IpcMessage const&, iox::runtime::IpcMessageType const&, iox::cxx::string<100ul> const&) <null> (iceoryx_integration_test+0x168bfd)
    #4 iox::roudi::RouDi::processRuntimeMessages() <null> (iceoryx_integration_test+0x162169)
    #5 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (iox::roudi::RouDi::*)(), iox::roudi::RouDi*> > >::_M_run() <null> (iceoryx_integration_test+0x169bd1)
    #6 <null> <null> (libstdc++.so.6+0xdc2b2)

  Previous read of size 1 at 0x7b6000000804 by main thread:
    #0 strnlen ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:400 (libtsan.so.0+0x3f2ed)
    #1 iox::posix::PosixUser::getUserName(unsigned int) <null> (iceoryx_integration_test+0x25d73c)
    #2 iox::posix::PosixUser::PosixUser(unsigned int) <null> (iceoryx_integration_test+0x25e8e6)
    #3 iox::posix::PosixUser::getUserOfCurrentProcess() <null> (iceoryx_integration_test+0x25ed49)
    #4 iox::runtime::IpcRuntimeInterface::IpcRuntimeInterface(iox::cxx::string<100ul> const&, iox::cxx::string<100ul> const&, iox::units::Duration) <null> (iceoryx_integration_test+0x2388
38)
    #5 iox::runtime::PoshRuntimeImpl::PoshRuntimeImpl(iox::cxx::optional<iox::cxx::string<100ul> const*>, iox::runtime::RuntimeLocation) <null> (iceoryx_integration_test+0x2191b5)
    #6 iox::runtime::PoshRuntimeSingleProcess::PoshRuntimeSingleProcess(iox::cxx::string<100ul> const&) <null> (iceoryx_integration_test+0x229bac)
<GTEST Stuff...>

  Location is heap block of size 1024 at 0x7b6000000800 allocated by main thread:
    #0 malloc ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:655 (libtsan.so.0+0x31c57)
    #1 getpwuid ../nss/getXXbyYY.c:121 (libc.so.6+0xe9712)
    #2 iox::posix::PosixUser::getUserName(unsigned int) <null> (iceoryx_integration_test+0x25d708)
    #3 iox::posix::PosixUser::PosixUser(unsigned int) <null> (iceoryx_integration_test+0x25e8e6)
    #4 iox::posix::PosixUser::getUserOfCurrentProcess() <null> (iceoryx_integration_test+0x25ed49)
    #5 iox::runtime::IpcRuntimeInterface::IpcRuntimeInterface(iox::cxx::string<100ul> const&, iox::cxx::string<100ul> const&, iox::units::Duration) <null> (iceoryx_integration_test+0x2388
38)
    #6 iox::runtime::PoshRuntimeImpl::PoshRuntimeImpl(iox::cxx::optional<iox::cxx::string<100ul> const*>, iox::runtime::RuntimeLocation) <null> (iceoryx_integration_test+0x2191b5)
    #7 iox::runtime::PoshRuntimeSingleProcess::PoshRuntimeSingleProcess(iox::cxx::string<100ul> const&) <null> (iceoryx_integration_test+0x229bac)
<GTEST Stuff...>

  Thread T11 'IPC-msg-process' (tid=50859, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (libstdc++.so.6+0xdc388)
    #2 iox::roudi::RouDi::RouDi(iox::roudi::RouDiMemoryInterface&, iox::roudi::PortManager&, iox::roudi::RouDi::RoudiStartupParameters) <null> (iceoryx_integration_test+0x164d80)
    #3 adp::sim::ipc::iceoryx::RoudiController::Start(std::vector<adp::sim::ipc::iceoryx::MemPoolConfig, std::allocator<adp::sim::ipc::iceoryx::MemPoolConfig> > const&) <null> (iceoryx_in
tegration_test+0x9bb76)
    #4 adp::sim::ipc::iceoryx::IceoryxIntegrationTest::IceoryxIntegrationTest() <null> (iceoryx_integration_test+0x304fa)
<GTEST Stuff...>

The test code is roughly along the lines of

class IceoryxIntegrationTest : public testing::Test {
 protected:
  IceoryxIntegrationTest() { roudi_controller_.Start({{kTestChunkByteCount, kNumChunks}}); }
  RoudiController roudi_controller_;
};

TEST_F(IceoryxIntegrationTest, CanPublishAndReceiveMessages) {
  // Precondition
  iox::runtime::PoshRuntimeSingleProcess runtime("IceoryxIntegrationTest");
  IceoryxUntypedPublisher publisher{{}};
  IceoryxUntypedSubscriber subscriber{{}};

...

[elBoberido]

@mcdougallryan while we have enabled TSAN builds, we got distracted halfway of making use of them. There is #692 waiting to be finished. What I recall from the PR when the TSAN build was introduced is that there were three false positives in iceoryx_hoofs test due to intentionally using a mutex in a wrong way to test an abstraction. In iceoryx_posh there were more hits but we did not have the time to fully investigate it. From a brief look at them it looked also mostly kind of false positives. We basically suppressed all TSAN warnings and postponed the investigation of the issues to a later time. This is definitely something we need to do before the v3 release.

In your specific case it is not necessarily a false positive. The getUserName method is using getpwuid which is does not need to be thread-safe according to the documentation. The documentation also suggest to use getpwuid_r as thread-safe alternative. I would need to have a closer look at the code base in order to determine if it is indeed a problem in the current code, i.e. this is indeed called from multiple threads but the chances are not low.

Do you want to contribute with a fix?

@elfenpiff what do you think, shall I create an issue to look at all posixCall invocations and check for re-entrant alternatives, e.g. getpwuid vs getpwuid_r?