For robots to robustly understand and interact with the physical world, it is highly beneficial to have a comprehensive representation - modelling geometry, physics, and visual observations - that informs perception, planning, and control algorithms. We propose a novel dual Gaussian-Particle representation that models the physical world while (i) enabling predictive simulation of future states and (ii) allowing online correction from visual observations in a dynamic world. Our representation comprises particles that capture the geometrical aspect of objects in the world and can be used alongside a particle-based physics system to anticipate physically plausible future states. Attached to these particles are 3D Gaussians that render images from any viewpoint through a splatting process thus capturing the visual state. By comparing the predicted and observed images, our approach generates visual forces that correct the particle positions while respecting known physical constraints. By integrating predictive physical modelling with continuous visually-derived corrections, our unified representation reasons about the present and future while synchronizing with reality. Our system runs in realtime at 30Hz using only 3 cameras. We validate our approach on 2D and 3D tracking tasks as well as photometric reconstruction quality.
为了使机器人能够稳健地理解和与物理世界互动，拥有一个全面的表示——模拟几何、物理和视觉观察——对于信息感知、规划和控制算法非常有益。我们提出了一种新颖的双高斯-粒子表示法，该表示法能够模拟物理世界，同时（i）实现未来状态的预测模拟和（ii）允许在动态世界中根据视觉观察进行在线校正。我们的表示包括捕捉世界中对象的几何方面的粒子，并可以与基于粒子的物理系统一起使用，以预测物理上可行的未来状态。这些粒子附加有3D高斯，通过平涂过程从任何视角渲染图像，从而捕捉视觉状态。通过比较预测图像和观察图像，我们的方法生成视觉力，这些视觉力在尊重已知物理约束的同时，校正粒子位置。通过将预测物理建模与连续的视觉导出校正相结合，我们的统一表示法在与现实同步的同时，对当前和未来进行推理。我们的系统仅使用3个摄像头即可实时运行，频率为30Hz。我们在2D和3D跟踪任务以及光度重建质量上验证了我们的方法。