We present a novel differentiable point-based rendering framework for material and lighting decomposition from multi-view images, enabling editing, ray-tracing, and real-time relighting of the 3D point cloud. Specifically, a 3D scene is represented as a set of relightable 3D Gaussian points, where each point is additionally associated with a normal direction, BRDF parameters, and incident lights from different directions. To achieve robust lighting estimation, we further divide incident lights of each point into global and local components, as well as view-dependent visibilities. The 3D scene is optimized through the 3D Gaussian Splatting technique while BRDF and lighting are decomposed by physically-based differentiable rendering. Moreover, we introduce an innovative point-based ray-tracing approach based on the bounding volume hierarchy for efficient visibility baking, enabling real-time rendering and relighting of 3D Gaussian points with accurate shadow effects. Extensive experiments demonstrate improved BRDF estimation and novel view rendering results compared to state-of-the-art material estimation approaches. Our framework showcases the potential to revolutionize the mesh-based graphics pipeline with a relightable, traceable, and editable rendering pipeline solely based on point cloud.
我们提出了一种新颖的可微分点基渲染框架，用于从多视图图像中进行材质和光照分解，使得3D点云的编辑、光线追踪和实时重新照明成为可能。具体来说，一个3D场景被表示为一组可重新照明的3D高斯点，其中每个点额外关联有法线方向、BRDF参数和来自不同方向的入射光。为了实现稳健的光照估计，我们进一步将每个点的入射光分为全局和局部组成部分，以及视角依赖的可见性。3D场景通过3D高斯飞溅技术进行优化，而BRDF和光照通过基于物理的可微分渲染进行分解。此外，我们引入了一种基于边界体积层次结构的创新点基光线追踪方法，用于高效的可见性烘焙，使得3D高斯点的实时渲染和重新照明能够实现准确的阴影效果。广泛的实验展示了与最先进的材质估计方法相比，我们的框架在BRDF估计和新视角渲染结果方面的改进。我们的框架展示了用基于点云的可重新照明、可追踪和可编辑渲染管线革命性地替代基于网格的图形管线的潜力。