INTEGRATION_distortion_camera seg-faults in gz_sim-ci-gz-sim8-jammy-amd64

[scpeters]

Environment

• OS Version: Ubuntu 22.04
• Source or binary build? source build of gz-sim8 in CI, using binaries for dependencies
• If this is a GUI or sensor rendering bug, describe your GPU and rendering system. Otherwise delete this section.
  • Rendering plugin: [ogre | ogre2].
    • Sensor rendering error.
    • GUI rendering error.
  • EGL headless mode:
    • Running in EGL headless mode
  • Generally, mention all circumstances that might affect rendering capabilities:
    • running on a dual GPU machine (integrated GPU + discrete GPU)
    • running on a multi-GPU machine (it has multiple discrete GPUs)
    • running on real hardware
    • running in virtual machine
    • running in Docker/Singularity
    • running remotely (e.g. via SSH)
    • running in a cloud
    • using VirtualGL, XVFB, Xdummy, XVNC or other indirect rendering utilities
    • GPU is concurrently used for other tasks
      • desktop acceleration
      • video decoding (i.e. a playing Youtube video)
      • video encoding
      • CUDA/ROCm computations (Tensorflow, Torch, Caffe running)
      • multiple simulators running at the same time
    • other...
  • Rendering system info:
    • On Linux, provide the outputs of the following commands:

      LANG=C lspci -nn | grep VGA  # might require installing pciutils
      echo "$DISPLAY"
      LANG=C glxinfo -B | grep -i '\(direct rendering\|opengl\|profile\)'  # might require installing mesa-utils package
      ps aux | grep Xorg
      sudo env LANG=C X -version  # if you don't have root access, try to tell the version of Xorg e.g. via package manager

    • On Windows, run dxdiag and report the GPU-related information.
    • On Mac OS, open a terminal and type system_profiler SPDisplaysDataType. Copy the output here.
  • Please, attach the ogre.log or ogre2.log file from ~/.gz/rendering

# paste log here

Description

• Expected behavior: INTEGRATION_distortion_camera should pass in CI
• Actual behavior: it fails in the gz_sim-ci-gz-sim8-jammy-amd64 Jenkins build

Steps to reproduce

1. Run the https://build.osrfoundation.org/job/gz_sim-ci-gz-sim8-jammy-amd64 build
2. Check the results of the INTEGRATION_distortion_camera test

Output

From the following build:

• https://build.osrfoundation.org/job/gz_sim-ci-gz-sim8-jammy-amd64/36/

        Start 269: INTEGRATION_distortion_camera

269: Test command: /home/jenkins/workspace/gz_sim-ci-gz-sim8-jammy-amd64/build/bin/INTEGRATION_distortion_camera "--gtest_output=xml:/home/jenkins/workspace/gz_sim-ci-gz-sim8-jammy-amd64/build/test_results/INTEGRATION_distortion_camera.xml"
269: Environment variables: 
269:  GZ_SIM_INSTALL_PREFIX=/usr
269: Test timeout computed to be: 240
269: Running main() from /home/jenkins/workspace/gz_sim-ci-gz-sim8-jammy-amd64/gz-sim/test/gtest_vendor/src/gtest_main.cc
269: [==========] Running 1 test from 1 test suite.
269: [----------] Global test environment set-up.
269: [----------] 1 test from DistortionCameraTest
269: [ RUN      ] DistortionCameraTest.DistortionCameraBox
269: [Msg] Loading SDF world file[/home/jenkins/workspace/gz_sim-ci-gz-sim8-jammy-amd64/gz-sim/test/worlds/camera_distortion.sdf].
269: [Msg] Serving entity system service on [/entity/system/add]
269: [Dbg] [Physics.cc:860] Loaded [gz::physics::dartsim::Plugin] from library [/usr/lib/x86_64-linux-gnu/gz-physics-7/engine-plugins/libgz-physics-dartsim-plugin.so]
269: [Dbg] [SystemManager.cc:74] Loaded system [gz::sim::systems::Physics] for entity [1]
269: [Dbg] [SystemManager.cc:74] Loaded system [gz::sim::systems::SceneBroadcaster] for entity [1]
269: [Dbg] [Sensors.cc:527] Configuring Sensors system
269: [Dbg] [Sensors.cc:446] SensorsPrivate::Run
269: [Dbg] [SystemManager.cc:74] Loaded system [gz::sim::systems::Sensors] for entity [1]
269: [Dbg] [Sensors.cc:422] SensorsPrivate::RenderThread started
269: [Dbg] [Sensors.cc:262] Waiting for init
269: [Msg] Loaded level [3]
269: [Msg] Serving world controls on [/world/lights/control], [/world/lights/control/state] and [/world/lights/playback/control]
269: [Msg] Serving GUI information on [/world/lights/gui/info]
269: [Msg] World [lights] initialized with [1ms] physics profile.
269: [Msg] Serving world SDF generation service on [/world/lights/generate_world_sdf]
269: [Msg] Serving world names on [/gazebo/worlds]
269: [Msg] Resource path add service on [/gazebo/resource_paths/add].
269: [Msg] Resource path get service on [/gazebo/resource_paths/get].
269: [Msg] Resource path resolve service on [/gazebo/resource_paths/resolve].
269: [Msg] Resource paths published on [/gazebo/resource_paths].
269: [Msg] Server control service on [/server_control].
269: [Msg] Found no publishers on /stats, adding root stats topic
269: [Msg] Found no publishers on /clock, adding root clock topic
269: [Dbg] [SimulationRunner.cc:545] Creating PostUpdate worker threads: 3
269: [Dbg] [SimulationRunner.cc:556] Creating postupdate worker thread (0)
269: [Dbg] [SimulationRunner.cc:556] Creating postupdate worker thread (1)
269: [Dbg] [Sensors.cc:700] Initialization needed
269: [Dbg] [Sensors.cc:274] Initializing render context
269: [Msg] Loading plugin [gz-rendering-ogre]
269: [Msg] Serving scene information on [/world/lights/scene/info]
269: [Msg] Serving graph information on [/world/lights/scene/graph]
269: [Msg] Serving full state on [/world/lights/state]
269: [Msg] Serving full state (async) on [/world/lights/s
269/320 Test #269: INTEGRATION_distortion_camera ...........................***Exception: SegFault  2.95 sec

[azeey]

I've been trying to generate a backtrace for this here without much luck. @iche033 hasn't been able to reproduce this locally, AFAIK.

[scpeters]

it doesn't fail on the gz-sim7 20.04 job:

• https://build.osrfoundation.org/job/gz_sim-ci-gz-sim7-focal-amd64/5/testReport/(root)/DistortionCameraTest/

[scpeters]

we should consider disabling this test in our Ubuntu Jenkins build since I think it is the only thing keeping gz_sim-ci-gz-sim8-jammy-amd64 from 🟢

[azeey]

we should consider disabling this test in our Ubuntu Jenkins build since I think it is the only thing keeping gz_sim-ci-gz-sim8-jammy-amd64 from 🟢

Yes, let's disable it. We'll keep this issue assigned so we won't forget about it.

[iche033]

the INTEGRATION_distortion_camera uses ogre 1.x. I think there are other shutdown issues in gz-rendering-ogre that may need to be fixed in addition to distortion. So disabling the test maybe a good option for now.

[scpeters]

apparently the test was already disabled for macOS and doesn't run on windows. I'll just outright disable it

[scpeters]

disabling test on Linux in #2247