Reconstructing controllable Gaussian splats from monocular video is a challenging task due to its inherently insufficient constraints. Widely adopted approaches supervise complex interactions with additional masks and control signal annotations, limiting their real-world applications. In this paper, we propose an annotation guidance-free method, dubbed FreeGaussian, that mathematically derives dynamic Gaussian motion from optical flow and camera motion using novel dynamic Gaussian constraints. By establishing a connection between 2D flows and 3D Gaussian dynamic control, our method enables self-supervised optimization and continuity of dynamic Gaussian motions from flow priors. Furthermore, we introduce a 3D spherical vector controlling scheme, which represents the state with a 3D Gaussian trajectory, thereby eliminating the need for complex 1D control signal calculations and simplifying controllable Gaussian modeling. Quantitative and qualitative evaluations on extensive experiments demonstrate the state-of-the-art visual performance and control capability of our method.
从单目视频中重建可控的高斯点是一项具有挑战性的任务，因为其内在的约束不足。广泛采用的方法通过额外的掩码和控制信号注释来监督复杂的交互，这在很大程度上限制了其在真实场景中的应用。本文提出了一种无需注释指导的方法，称为FreeGaussian。该方法通过创新的动态高斯约束，从光流和相机运动中数学推导出动态高斯运动。通过建立二维光流和三维高斯动态控制之间的联系，我们的方法实现了从流先验中自监督优化和动态高斯运动的连续性。此外，我们引入了一种三维球面向量控制方案，通过三维高斯轨迹表示状态，从而避免了复杂的一维控制信号计算，简化了可控高斯建模。大量实验中的定量和定性评估结果表明，我们的方法在视觉效果和控制能力方面达到了最新的性能水准。