In this work, we propose a novel method to supervise 3D Gaussian Splatting (3DGS) scenes using optical tactile sensors. Optical tactile sensors have become widespread in their use in robotics for manipulation and object representation; however, raw optical tactile sensor data is unsuitable to directly supervise a 3DGS scene. Our representation leverages a Gaussian Process Implicit Surface to implicitly represent the object, combining many touches into a unified representation with uncertainty. We merge this model with a monocular depth estimation network, which is aligned in a two stage process, coarsely aligning with a depth camera and then finely adjusting to match our touch data. For every training image, our method produces a corresponding fused depth and uncertainty map. Utilizing this additional information, we propose a new loss function, variance weighted depth supervised loss, for training the 3DGS scene model. We leverage the DenseTact optical tactile sensor and RealSense RGB-D camera to show that combining touch and vision in this manner leads to quantitatively and qualitatively better results than vision or touch alone in a few-view scene syntheses on opaque as well as on reflective and transparent objects.
在这项工作中，我们提出了一种利用光学触觉传感器监督3D高斯溅射（3DGS）场景的新方法。光学触觉传感器在机器人操控和物体表征方面的使用已经变得广泛；然而，原始的光学触觉传感器数据不适合直接用于监督3DGS场景。我们的表示利用高斯过程隐式曲面来隐式表示对象，将许多触摸合并成一个具有不确定性的统一表示。我们将这个模型与单眼深度估计网络合并，通过一个两阶段过程进行对齐，首先粗略地与深度摄像机对齐，然后精细调整以匹配我们的触觉数据。对于每一张训练图像，我们的方法产生一个相应的融合深度和不确定性地图。利用这些额外信息，我们提出了一个新的损失函数，方差加权深度监督损失，用于训练3DGS场景模型。我们利用DenseTact光学触觉传感器和RealSense RGB-D摄像机展示，以这种方式结合触觉和视觉，相比单独使用视觉或触觉，在少视角场景合成上，无论是对于不透明物体还是反射和透明物体，都能得到定量和定性更好的结果。