HybridGS: Decoupling Transients and Statics with 2D and 3D Gaussian Splatting

Generating high-quality novel view renderings of 3D Gaussian Splatting (3DGS) in scenes featuring transient objects is challenging. We propose a novel hybrid representation, termed as HybridGS, using 2D Gaussians for transient objects per image and maintaining traditional 3D Gaussians for the whole static scenes. Note that, the 3DGS itself is better suited for modeling static scenes that assume multi-view consistency, but the transient objects appear occasionally and do not adhere to the assumption, thus we model them as planar objects from a single view, represented with 2D Gaussians. Our novel representation decomposes the scene from the perspective of fundamental viewpoint consistency, making it more reasonable. Additionally, we present a novel multi-view regulated supervision method for 3DGS that leverages information from co-visible regions, further enhancing the distinctions between the transients and statics. Then, we propose a straightforward yet effective multi-stage training strategy to ensure robust training and high-quality view synthesis across various settings. Experiments on benchmark datasets show our state-of-the-art performance of novel view synthesis in both indoor and outdoor scenes, even in the presence of distracting elements.

Given a casually captured image sequence, we decompose the whole scene into 2D Gaussians for transient objects and 3D Gaussians for static scenes. To warm up, we start by training a basic 3DGS to capture static elements. This is followed by iterative training of 2D and 3D Gaussians, where our transients and statics are combined using an \(\alpha\)-blending strategy with masks to produce the final renderings. The masks provide guidance for 3D Gaussians in the iterative training stage. During the joint-training, both 2D and 3D Gaussians are trained to further optimize the decomposition results.

Essentially, given a set of input images \(\{I_k | k=1,2,..., N\}\) with the corresponding camera parameters, for each view \(I\), the goal of our method is to reasonably decouple the transients \(I_t\) and statics \(I_s\) as follows:

\[ I = M_t \odot I_t + (1 - M_t) \odot I_s, \]

where the \(M_t \in [0, 1]\) represents the transient mask, and \(\odot\) is per pixel multiplication. When a pixel's value in \(M_t\) approaches 1, it indicates a higher probability that the location is transient; conversely, a lower value suggests a greater likelihood that area is static.

To achieve the goal, we decompose the whole 3D scene with two components:
(1) Multi-view consistent 3D Gaussians is used for rendering \(I_s\), which leverages multi-view information from images and models the static scene with a set of unified 3D Gaussians. The multi-view inputs regulate the 3D Gaussians to be consistent and robust across the different views.
(2) Single-view independent 2D Gaussians is responsible for modeling \(I_t\), which enables our approach to handle the fact that images are casually captured with varying transients. Concretely, we form a set of view-independent 2D Gaussians to model transients as planar objects from a single view.

This novel combination allows for a more precise and reasonable representation of 3D scenes, enabling better performance on novel views.

Our method demonstrates superior results by effectively reducing artifacts and providing clearer boundaries. This results in a cleaner statics compared to other methods, showcasing enhanced visual quality and precision in novel views.

Also, our method achieves superior transient mask separation in both indoor and outdoor scenes. It effectively separates transients and statics, and the resulting renderings closely resemble the ground truth images, demonstrating its effectiveness.

Finally, as training iterations increase, 3DGS tends to gradually integrate transient elements into the static components, rendering the residuals being almost incapable of capturing transient contents. In contrast, our HybridGS effectively distinguishes transients from statics over time, leading to consistent improvements.