Real2Sim2Real plays a critical role in robotic arm control and reinforcement learning, yet bridging this gap remains a significant challenge due to the complex physical properties of robots and the objects they manipulate. Existing methods lack a comprehensive solution to accurately reconstruct real-world objects with spatial representations and their associated physics attributes. We propose a Real2Sim pipeline with a hybrid representation model that integrates mesh geometry, 3D Gaussian kernels, and physics attributes to enhance the digital asset representation of robotic arms. This hybrid representation is implemented through a Gaussian-Mesh-Pixel binding technique, which establishes an isomorphic mapping between mesh vertices and Gaussian models. This enables a fully differentiable rendering pipeline that can be optimized through numerical solvers, achieves high-fidelity rendering via Gaussian Splatting, and facilitates physically plausible simulation of the robotic arm's interaction with its environment using mesh-based methods.
Real2Sim2Real在机器人臂控制和强化学习中扮演着关键角色，但由于机器人及其操作的物体的复杂物理属性，弥合这一差距仍然是一个重大挑战。现有方法缺乏一种全面的解决方案来准确重建具有空间表示和相关物理属性的真实世界物体。 我们提出了一种Real2Sim管道，采用混合表示模型，将网格几何、3D高斯核和物理属性结合起来，以增强机器人臂的数字资产表示。这种混合表示通过高斯-网格-像素绑定技术实现，该技术在网格顶点和高斯模型之间建立了同构映射。这使得通过数值求解器进行优化的完全可微分渲染管道成为可能，通过高斯斑点实现高保真渲染，并使用基于网格的方法促进机器人臂与环境交互的物理上合理的模拟。