The recent Gaussian Splatting achieves high-quality and real-time novel-view synthesis of the 3D scenes. However, it is solely concentrated on the appearance and geometry modeling, while lacking in fine-grained object-level scene understanding. To address this issue, we propose Gaussian Grouping, which extends Gaussian Splatting to jointly reconstruct and segment anything in open-world 3D scenes. We augment each Gaussian with a compact Identity Encoding, allowing the Gaussians to be grouped according to their object instance or stuff membership in the 3D scene. Instead of resorting to expensive 3D labels, we supervise the Identity Encodings during the differentiable rendering by leveraging the 2D mask predictions by SAM, along with introduced 3D spatial consistency regularization. Comparing to the implicit NeRF representation, we show that the discrete and grouped 3D Gaussians can reconstruct, segment and edit anything in 3D with high visual quality, fine granularity and efficiency. Based on Gaussian Grouping, we further propose a local Gaussian Editing scheme, which shows efficacy in versatile scene editing applications, including 3D object removal, inpainting, colorization and scene recomposition.
最近的高斯喷溅技术实现了3D场景的高质量和实时新视图合成。然而，它仅专注于外观和几何建模，而缺乏细粒度的对象级场景理解。为了解决这个问题，我们提出了高斯分组，这是对高斯喷溅的扩展，用于同时重建和分割开放世界3D场景中的任何事物。我们为每个高斯增加了一个紧凑的身份编码，允许根据3D场景中的对象实例或材料成员将高斯进行分组。我们不是求助于昂贵的3D标签，而是在可微渲染过程中通过利用SAM的2D遮罩预测来监督身份编码，同时引入了3D空间一致性正则化。与隐式的NeRF表示相比，我们展示了离散且分组的3D高斯可以以高视觉质量、细粒度和效率在3D中重建、分割和编辑任何事物。基于高斯分组，我们进一步提出了一种局部高斯编辑方案，该方案在多种场景编辑应用中显示出有效性，包括3D对象移除、修复、上色和场景重组。