Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data races in drt module #6187

Open
eszpotanski opened this issue Nov 18, 2024 · 0 comments
Open

Data races in drt module #6187

eszpotanski opened this issue Nov 18, 2024 · 0 comments
Assignees
@osamahammad21
Labels
drt Detailed Routing

Comments

@eszpotanski
Copy link
Contributor

Describe the bug

When running 5_2_route stage for Ibex, ThreadSanitizer detects a few data races, usually in drt::FlexPA and drt::FlexTA.

Expected Behavior

Successful run without data races

Environment

[WARNING] Your current OpenROAD version is outdated.
It is recommened to pull the latest changes.
If problem persists, file a github issue with the re-producible test case.
kernel: Linux 6.11.5-arch1-1
os: Arch Linux
cmake version 3.30.5
-- The CXX compiler identification is GNU 14.2.1
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- OpenROAD version: v2.0-17013-gf7f634f88
-- System name: Linux
-- Compiler: GNU 14.2.1
-- Build type: RELEASE
-- Install prefix: /usr/local
-- C++ Standard: 17
-- C++ Standard Required: ON
-- C++ Extensions: OFF
-- The C compiler identification is GNU 14.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Performing Test C_COMPILER_SUPPORTS__-Wall
-- Performing Test C_COMPILER_SUPPORTS__-Wall - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wall
-- Performing Test CXX_COMPILER_SUPPORTS__-Wall - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-array-bounds
-- Performing Test C_COMPILER_SUPPORTS__-Wno-array-bounds - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-array-bounds
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-array-bounds - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-nonnull
-- Performing Test C_COMPILER_SUPPORTS__-Wno-nonnull - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-nonnull
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-nonnull - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-maybe-uninitialized
-- Performing Test C_COMPILER_SUPPORTS__-Wno-maybe-uninitialized - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-maybe-uninitialized
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-maybe-uninitialized - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format-overflow
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format-overflow - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format-overflow
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format-overflow - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-variable
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-variable - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-variable
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-variable - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-function
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-function - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-function
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-function - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-write-strings
-- Performing Test C_COMPILER_SUPPORTS__-Wno-write-strings - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-write-strings
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-write-strings - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-sign-compare
-- Performing Test C_COMPILER_SUPPORTS__-Wno-sign-compare - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-sign-compare
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-sign-compare - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-deprecated
-- Performing Test C_COMPILER_SUPPORTS__-Wno-deprecated - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-deprecated
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-deprecated - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-c++11-narrowing
-- Performing Test C_COMPILER_SUPPORTS__-Wno-c++11-narrowing - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-c++11-narrowing
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-c++11-narrowing - Failed
-- Performing Test C_COMPILER_SUPPORTS__-Wno-register
-- Performing Test C_COMPILER_SUPPORTS__-Wno-register - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-register
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-register - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal
-- PerformingCMake Warning (dev) at src/CMakeLists.txt:175 (find_package):
  Policy CMP0167 is not set: The FindBoost module is removed.  Run "cmake
  --help-policy CMP0167" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning at src/CMakeLists.txt:267 (message):
  spdlog: SPDLOG_FMT_EXTERNAL=ON


CMake Warning (dev) at src/dpl/CMakeLists.txt:41 (find_package):
  Policy CMP0167 is not set: The FindBoost module is removed.  Run "cmake
  --help-policy CMP0167" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at src/cts/CMakeLists.txt:36 (find_package):
  Policy CMP0167 is not set: The FindBoost module is removed.  Run "cmake
  --help-policy CMP0167" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at src/drt/CMakeLists.txt:49 (find_package):
  Policy CMP0167 is not set: The FindBoost module is removed.  Run "cmake
  --help-policy CMP0167" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at src/utl/CMakeLists.txt:38 (find_package):
  Policy CMP0167 is not set: The FindBoost module is removed.  Run "cmake
  --help-policy CMP0167" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

This warning is for project developers.  Use -Wno-dev to suppress it.

 Test C_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal - Failed
-- Performing Test C_COMPILER_SUPPORTS__-fpermissive
-- Performing Test C_COMPILER_SUPPORTS__-fpermissive - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-fpermissive
-- Performing Test CXX_COMPILER_SUPPORTS__-fpermissive - Success
-- Performing Test C_COMPILER_SUPPORTS__-x
-- Performing Test C_COMPILER_SUPPORTS__-x - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-x
-- Performing Test CXX_COMPILER_SUPPORTS__-x - Failed
-- Performing Test C_COMPILER_SUPPORTS__c++
-- Performing Test C_COMPILER_SUPPORTS__c++ - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__c++
-- Performing Test CXX_COMPILER_SUPPORTS__c++ - Failed
-- Performing Test C_COMPILER_SUPPORTS__-std=c++17
-- Performing Test C_COMPILER_SUPPORTS__-std=c++17 - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-std=c++17
-- Performing Test CXX_COMPILER_SUPPORTS__-std=c++17 - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-but-set-variable
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-but-set-variable - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-but-set-variable
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-but-set-variable - Success
-- TCL library: /usr/lib/libtcl.so
-- TCL header: /usr/include/tcl.h
-- TCL readline library: /usr/lib/libtclreadline.so
-- TCL readline header: /usr/include
-- Found SWIG: /usr/bin/swig (found suitable version "4.2.1", minimum required is "4.0")
-- Using SWIG >= 4.2.1 -flatstaticmethod flag for python
-- Found Boost: /usr/lib/cmake/Boost-1.86.0/BoostConfig.cmake (found version "1.86.0")
-- boost: 1.86.0
-- Found GTest: /usr/lib/cmake/GTest/GTestConfig.cmake (found version "1.15.2")
-- GTest: 1.15.2
-- Found Python3: /usr/include/python3.12 (found version "3.12.7") found components: Development Development.Module Development.Embed
-- Found ZLIB: /usr/lib/libz.so (found version "1.3.1")
-- Found Threads: TRUE
-- spdlog: 1.14.1
-- Found BISON: /usr/bin/bison (found version "3.8.2")
-- Found Doxygen: /usr/bin/doxygen (found version "1.12.0") found components: doxygen dot
-- STA version: 2.6.0
-- STA git sha: aa598a2f14c5c142e90391a69988523505e7db3d
-- System name: Linux
-- Compiler: GNU 14.2.1
-- Build type: RELEASE
-- Build CXX_FLAGS: -O3 -DNDEBUG
-- Install prefix: /usr/local
-- Found FLEX: /usr/bin/flex (found version "2.6.4")
-- TCL library: /usr/lib/libtcl.so
-- TCL header: /usr/include/tcl.h
-- TCL readline library: /usr/lib/libtclreadline.so
-- TCL readline header: /usr/include/tclreadline.h
-- CUDD library: /usr/local/lib/libcudd.a
-- CUDD header: /usr/local/include/cudd.h
-- SSTA: 0
-- Found SWIG: /usr/bin/swig (found suitable version "4.2.1", minimum required is "3.0")
-- STA executable: /home/ARCHIVE/OPENROAD/google-openroad-flow-scripts/tools/OpenROAD/src/sta/app/sta
-- Found re2: /opt/or-tools/lib/cmake/re2/re2Config.cmake (found version "11.0.0")
-- Found Clp: /opt/or-tools/lib/cmake/Clp/ClpConfig.cmake (found version "1.17.7")
-- Found Cbc: /opt/or-tools/lib/cmake/Cbc/CbcConfig.cmake (found version "2.10.7")
-- Found SCIP: /opt/or-tools/lib/cmake/scip/scip-config.cmake (found version "9.0.0")
-- Found OpenMP_CXX: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- Found OR-Tools: /opt/or-tools/lib/cmake/ortools (version: 9.11.4210)
-- TCL library: /usr/lib/libtcl.so
-- TCL header: /usr/include/tcl.h
-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- GUI is enabled
-- Charts widget is not enabled
-- Found Boost: /usr/lib/cmake/Boost-1.86.0/BoostConfig.cmake (found version "1.86.0") found components: serialization
-- Could NOT find VTune (missing: VTune_LIBRARIES VTune_INCLUDE_DIRS)
-- Found Boost: /usr/lib/cmake/Boost-1.86.0/BoostConfig.cmake (found suitable version "1.86.0", minimum required is "1.78")
-- TCL library: /usr/lib/libtcCMake Warning (dev) at src/dst/CMakeLists.txt:42 (find_package):
  Policy CMP0167 is not set: The FindBoost module is removed.  Run "cmake
  --help-policy CMP0167" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at src/dft/test/CMakeLists.txt:18 (find_package):
  Policy CMP0167 is not set: The FindBoost module is removed.  Run "cmake
  --help-policy CMP0167" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

This warning is for project developers.  Use -Wno-dev to suppress it.

Number of processor cores: 16
l.so
-- TCL header: /usr/include/tcl.h
-- Found Boost: /usr/lib/cmake/Boost-1.86.0/BoostConfig.cmake (found version "1.86.0") found components: serialization system thread
-- Found Boost: /usr/lib/cmake/Boost-1.86.0/BoostConfig.cmake (found version "1.86.0")
-- Found Eigen3: /usr/share/eigen3/cmake/Eigen3Config.cmake (found version "3.4.0")
-- TCL readline enabled
-- Tcl Extended disabled
-- Python3 enabled
-- Configuring done (9.5s)
-- Generating done (0.8s)
-- Build files have been written to: /tmp/tmp.pvRoKTV8Mz

To Reproduce

Default flow with ASAP7 Ibex used. Data races can be reproduced with:

  • Run GRT for Ibex, just to prepare dependencies - DESIGN_CONFIG=designs/asap7/ibex/config.mk make grt (from OpenROAD-flow-scripts/flow)
  • Rebuild OpenROAD with TSAN option - ./build_openroad.sh --local --openroad-args " -DTSAN=ON " (from OpenROAD-flow-scripts)
  • Run 5_2_route for Ibex - DESIGN_CONFIG=designs/asap7/ibex/config.mk make do-5_2_route (from OpenROAD-flow-scripts/flow)

Relevant log output

==================
WARNING: ThreadSanitizer: data race (pid=69183)
  Read of size 8 at 0x7ffffb23f838 by thread T18:
    #0 drt::FlexPA::initAllAccessPoints() [clone ._omp_fn.0] /OpenROAD-flow-scripts/tools/OpenROAD/src/drt/src/pa/FlexPA_prep.cpp:1518 (openroad+0x1bc799c)
    #1 <null> <null> (libgomp.so.1+0x1cd8d)

  Previous write of size 8 at 0x7ffffb23f838 by main thread:
    #0 drt::FlexPA::initAllAccessPoints() /OpenROAD-flow-scripts/tools/OpenROAD/src/drt/src/pa/FlexPA_prep.cpp:1500 (openroad+0x1bb969d)
    #1 drt::FlexPA::prep() /OpenROAD-flow-scripts/tools/OpenROAD/src/drt/src/pa/FlexPA.cpp:112 (openroad+0x1b993d0)
    #2 drt::FlexPA::main() /OpenROAD-flow-scripts/tools/OpenROAD/src/drt/src/pa/FlexPA.cpp:185 (openroad+0xcd56ae)
    #3 drt::TritonRoute::main() /OpenROAD-flow-scripts/tools/OpenROAD/src/drt/src/TritonRoute.cpp:975 (openroad+0xcb6053)
    #4 detailed_route_cmd(char const*, char const*, char const*, char const*, char const*, bool, int, char const*, char const*, int, double, char const*, char const*, int, bool, bool, bool, int, bool, char const*, int) /OpenROAD-flow-scripts/tools/OpenROAD/build/src/drt/CMakeFiles/drt.dir/TritonRouteTCL_wrap.cxx:1842 (openroad+0x1801e57)
    #5 _wrap_detailed_route_cmd /OpenROAD-flow-scripts/tools/OpenROAD/build/src/drt/CMakeFiles/drt.dir/TritonRouteTCL_wrap.cxx:2546 (openroad+0x18025e8)
    #6 TclNRRunCallbacks <null> (libtcl8.6.so+0x420c7)
    #7 tclAppInit /OpenROAD-flow-scripts/tools/OpenROAD/src/Main.cc:507 (openroad+0xd0ce43)
    #8 ord::tclInit(Tcl_Interp*) /OpenROAD-flow-scripts/tools/OpenROAD/src/Main.cc:550 (openroad+0xd0d9ae)
    #9 ord::tclAppInit(Tcl_Interp*) /OpenROAD-flow-scripts/tools/OpenROAD/src/Main.cc:545 (openroad+0xd0d9ae)
    #10 Tcl_MainEx <null> (libtcl8.6.so+0x108aa0)
    #11 __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 (libc.so.6+0x27249)
[...]
==================
WARNING: ThreadSanitizer: data race (pid=69183)
  Read of size 8 at 0x7b340001b3c0 by thread T22:
    #0 std::__uniq_ptr_impl<drt::FlexTAWorkerRegionQuery::Impl, std::default_delete<drt::FlexTAWorkerRegionQuery::Impl> >::_M_ptr() const /usr/include/c++/12/bits/unique_ptr.h:191 (openroad+0x1c95142)
    #1 std::unique_ptr<drt::FlexTAWorkerRegionQuery::Impl, std::default_delete<drt::FlexTAWorkerRegionQuery::Impl> >::get() const /usr/include/c++/12/bits/unique_ptr.h:462 (openroad+0x1c95142)
    #2 std::unique_ptr<drt::FlexTAWorkerRegionQuery::Impl, std::default_delete<drt::FlexTAWorkerRegionQuery::Impl> >::operator->() const /usr/include/c++/12/bits/unique_ptr.h:455 (openroad+0x1c95142)
    #3 drt::FlexTAWorkerRegionQuery::getDesign() const /OpenROAD-flow-scripts/tools/OpenROAD/src/drt/src/ta/FlexTA_rq.cpp:58 (openroad+0x1c95142)
    #4 drt::FlexTAWorkerRegionQuery::init() /OpenROAD-flow-scripts/tools/OpenROAD/src/drt/src/ta/FlexTA_rq.cpp:114 (openroad+0x1c95142)
    #5 drt::FlexTAWorker::init() /OpenROAD-flow-scripts/tools/OpenROAD/src/drt/src/ta/FlexTA_init.cpp:895 (openroad+0x1c8cec6)
    #6 drt::FlexTAWorker::main_mt() /OpenROAD-flow-scripts/tools/OpenROAD/src/drt/src/ta/FlexTA.cpp:72 (openroad+0x1964c1c)
    #7 drt::FlexTA::initTA_helper(int, int, int, bool, int&) [clone ._omp_fn.0] /OpenROAD-flow-scripts/tools/OpenROAD/src/drt/src/ta/FlexTA.cpp:178 (openroad+0x19657e0)
    #8 <null> <null> (libgomp.so.1+0x1cd8d)

  Previous write of size 8 at 0x7b340001b3c0 by main thread:
    #0 operator new(unsigned long) ../../../../src/libsanitizer/tsan/tsan_new_delete.cpp:64 (libtsan.so.2+0x87323)
    #1 std::__detail::_MakeUniq<drt::FlexTAWorker>::__single_object std::make_unique<drt::FlexTAWorker, drt::frDesign*, utl::Logger*&, bool&>(drt::frDesign*&&, utl::Logger*&, bool&) /usr/include/c++/12/bits/unique_ptr.h:1065 (openroad+0x195f0ee)
    #2 drt::FlexTA::initTA_helper(int, int, int, bool, int&) /OpenROAD-flow-scripts/tools/OpenROAD/src/drt/src/ta/FlexTA.cpp:148 (openroad+0x195f0ee)
    #3 drt::FlexTA::initTA(int) /OpenROAD-flow-scripts/tools/OpenROAD/src/drt/src/ta/FlexTA.cpp:235 (openroad+0x1963164)
    #4 drt::FlexTA::main() /OpenROAD-flow-scripts/tools/OpenROAD/src/drt/src/ta/FlexTA.cpp:335 (openroad+0xcc90ec)
    #5 drt::TritonRoute::ta() /OpenROAD-flow-scripts/tools/OpenROAD/src/drt/src/TritonRoute.cpp:623 (openroad+0x179157a)
    #6 drt::TritonRoute::main() /OpenROAD-flow-scripts/tools/OpenROAD/src/drt/src/TritonRoute.cpp:1001 (openroad+0xcb5b8d)
    #7 detailed_route_cmd(char const*, char const*, char const*, char const*, char const*, bool, int, char const*, char const*, int, double, char const*, char const*, int, bool, bool, bool, int, bool, char const*, int) /OpenROAD-flow-scripts/tools/OpenROAD/build/src/drt/CMakeFiles/drt.dir/TritonRouteTCL_wrap.cxx:1842 (openroad+0x1801e57)
    #8 _wrap_detailed_route_cmd /OpenROAD-flow-scripts/tools/OpenROAD/build/src/drt/CMakeFiles/drt.dir/TritonRouteTCL_wrap.cxx:2546 (openroad+0x18025e8)
    #9 TclNRRunCallbacks <null> (libtcl8.6.so+0x420c7)
    #10 tclAppInit /OpenROAD-flow-scripts/tools/OpenROAD/src/Main.cc:507 (openroad+0xd0ce43)
    #11 ord::tclInit(Tcl_Interp*) /OpenROAD-flow-scripts/tools/OpenROAD/src/Main.cc:550 (openroad+0xd0d9ae)
    #12 ord::tclAppInit(Tcl_Interp*) /OpenROAD-flow-scripts/tools/OpenROAD/src/Main.cc:545 (openroad+0xd0d9ae)
    #13 Tcl_MainEx <null> (libtcl8.6.so+0x108aa0)
    #14 __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 (libc.so.6+0x27249)
[...]

Screenshots

No response

Additional Context

As ThreadSanitizer warnings can be quite long, logs are attached as file (it may not contain all data races, because process was terminated after a few hours): 5_2_route.log
@maliberty maliberty added the drt Detailed Routing label Nov 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@osamahammad21 osamahammad21

Labels
drt Detailed Routing
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@maliberty @osamahammad21 @eszpotanski