Gaussian-Flow: 4D Reconstruction with Dynamic 3D Gaussian Particle
We're currently working on this page. Please check back soon.
We're currently working on this page. Please check back soon.
We introduce Gaussian-Flow, a novel point-based approach for fast dynamic scene reconstruction and real-time rendering from both multi-view and monocular videos. In contrast to the prevalent NeRF-based approaches hampered by slow training and rendering speeds, our approach harnesses recent advancements in point-based 3D Gaussian Splatting (3DGS). Specifically, a novel Dual-Domain Deformation Model (DDDM) is proposed to explicitly model attribute deformations of each Gaussian point, where the time-dependent residual of each attribute is captured by a polynomial fitting in the time domain, and a Fourier series fitting in the frequency domain. The proposed DDDM is capable of modeling complex scene deformations across long video footage, eliminating the need for training separate 3DGS for each frame or introducing an additional implicit neural field to model 3D dynamics. Moreover, the explicit deformation modeling for discretized Gaussian points ensures ultra-fast training and rendering of a 4D scene, which is comparable to the original 3DGS designed for static 3D reconstruction. Our proposed approach showcases a substantial efficiency improvement, achieving a $5\times$ faster training speed compared to the per-frame 3DGS modeling. In addition, quantitative results demonstrate that the proposed Gaussian-Flow significantly outperforms previous leading methods in novel view rendering quality.
The proposed differentiable rendering pipeline. We model the deformation of attributes of each 3D Gaussian point independently by using the Dual-Domain Deformation Model (DDDM), which preserves the discretized nature of the 3D Gaussian points, and thus achieves ultra-fast training and rendering speed comparable with the original 3DGS.
| RGB | Depth | RGB | Depth |
View Synthesis Results and Depths on Plenoptic Video Dataset.
Acknowledgements: The website template was borrowed from Lior Yariv.