GaussReg: Fast 3D Registration with Gaussian Splatting

Abstract

Point cloud registration is a fundamental problem for large-scale 3D scene scanning and reconstruction. With the help of deep learning, registration methods have evolved significantly, reaching a nearly-mature stage. As the introduction of Neural Radiance Fields (NeRF), it has become the most popular 3D scene representation as its powerful view synthesis capabilities. Regarding NeRF representation, its registration is also required for large-scale scene reconstruction. However, this topic extremly lacks exploration. This is due to the inherent challenge to model the geometric relationship among two scenes with implicit representations. The existing methods usually convert the implicit representation to explicit representation for further registration. Most recently, Gaussian Splatting (GS) is introduced, employing explicit 3D Gaussian. This method significantly enhances rendering speed while maintaining high rendering quality. Given two scenes with explicit GS representations, in this work, we explore the 3D registration task between them. To this end, we propose GaussReg, a novel coarse-to-fine framework, both fast and accurate. The coarse stage follows existing point cloud registration methods and estimates a rough alignment for point clouds from GS. We further newly present an image-guided fine registration approach, which renders images from GS to provide more detailed geometric information for precise alignment. To support comprehensive evaluation, we carefully build a scene-level dataset called ScanNet-GSReg with $1379$ scenes obtained from the ScanNet dataset and collect an in-the-wild dataset called GSReg. Experimental results demonstrate our method achieves state-of-the-art performance on multiple datasets. Our GaussReg is $44\times$ faster than HLoc (SuperPoint as the feature extractor and SuperGlue as the matcher) with comparable accuracy.

1 Introduction

Figure 1: The purpose of our method is to register scenes A and B with Gaussian Splatting[16] models, and then combine A with B to get the fused Gaussian Splatting model. The first row is the visualization of the 3D Gaussians.

In traditional 3D scene scanning and reconstruction, a large-scale scene is usually divided into different blocks, resulting in many independent sub-scenes that may not in the same coordinate system. Therefore, the registration between them plays a crucial role. Currently, point cloud registration has been widely studied and reached a relative mature stage, with several representative works such as ICP [4], D3Feat [1], Geotransformer [26], etc. The mainstream methods typically involve extracting features from point clouds and locating matching points to calculate the transformation between the two input scenes.

Recently, a new 3D representation - Neural Radiance Fields (NeRF) has been introduced and quickly gained attention due to its powerful capability in view synthesis, and it has been widely used in representing 3D scenes. When considering large-scale scene reconstruction based on NeRF, there are two main challenges: 1) Due to the complex occlusions present in real-world scenes, lots of images or videos are often required to capture for large-scale reconstruction, leading to a time-consuming data collection process. 2) Optimizing NeRF with numerous images is computationally intensive. Therefore, a direct approach is to divide a large-scale scene into some smaller scenes, reconstruct them separately, and then use registration to combine all these small scenes together.

Consider two overlapping scenes, each with its own NeRF model. Currently, the methods for registering two reconstructed NeRF scenes can be generally categorized into two types: 1) As the method proposed in NeRFuser [10], we can render a large number of images for each scene, then recover poses of all these images together from structure-from-motion (SfM). However, this method is very time-consuming; 2) As in the method DReg-NeRF [6], we can convert the implicit radiance field to explicit voxel by querying voxel grids from NeRF of two scenes, and extract features to establish their matching relationship for registration. But this method faces two issues: a) it is difficult to turn NeRF of unbounded scene to bounded voxel; b) the resolution limitation of the voxel grid makes this method unsuitable for larger scenes.

Most recently, Gaussian Splatting (GS) [16] has been proposed, which introduces an explicit representation of 3D Gaussians, ensuring high-quality rendering while speeding up the rendering process. Then, an interesting question comes up: “As GS provides a point-like representation, can we conduct GS registration resorting to point-cloud registration methods? ”

In this work, we explore fast and accurate 3D registration with GS to answer the question. Taking GS models of the two scenes as input, we first extract their point clouds from GS. Thus, the straightforward approach is to adapt point cloud registration methods to the registration between these GS point clouds. To this end, a coarse registration method is designed which follows standard point cloud registration pipeline, such as GeoTransformer [26], but with special consideration of extra attributes (e.g., opacity) in 3D gaussians.

Compared with traditionally collected point cloud data, point clouds from GS only capture rough geometric structure and are usually noisy. Thus, a coarse registration can not achieve precise results with sufficient accuracy. We further propose a novel image-guided fine registration pipeline built upon the coarse registration result. Our main idea is from the observation that GS not only contains geometry information but also inherently detailed image information, which can support more accurate alignment. Therefore, we first locate the overlapping regions with the help of coarse registration, where a few images are rendered with the help of GS. Then, the fine registration pipeline projects images into 3D volumetric features for final matching and transformation estimation.

Ultimately, we propose a novel coarse-to-fine GS registration framework: GaussReg. However, it still lacks evaluation benchmarks of scene-level registration with GS. To support this, we construct a dataset called ScanNet-GSReg, comprising 1379 scenes from the ScanNet [7] dataset. In addition, we collect a dataset named GSReg, comprising 6 indoor and 4 outdoor scenarios, to assess the generalization capability of our method. We conduct extensive experiments on the ScanNet-GSReg dataset, the Objaverse [8] dataset used in DReg-NeRF [6], and the GSReg dataset, demonstrating the effectiveness of our method.

The main contributions can be summarized as:

• To the best of our knowledge, we are the first to explore the registration of 3D scenes considering Gaussian Splatting representations.

• We carefully designed a novel coarse-to-fine pipeline that fully considers the characteristics of 3D gaussians, which performs both fast and accurate.

• An image-guided fine registration is newly presented that takes rendered images of GS into account for fine-level alignment. We also believe this strategy opens minds fro GS-related researches.

• A benchmark is also newly built for the proposed new task, which includes scenes from ScanNet and several self-collected in-the-wild scenes.

2 Related Work

3D point cloud registration

3D point cloud registration has been developed for decades. Given two overlapping point clouds with different coordinate systems, the target of this task is to find the transformation between them. Traditional methods [4, 18, 17, 40, 21, 23, 42] divide this process into two parts: correspondence searching and transformation estimation. Correspondence searching involves finding sparse matched feature points between the source and target point clouds. Transformation estimation is to calculate the transformation matrix using these correspondences. These two stages will be conducted iteratively to find the optimal transformation. However, these methods require many complex strategies [18, 17, 23, 42] to overcome noise, outliers, or density variations. To overcome these problems, deep feature extractors [35, 13, 41] are proposed to find more robust correspondences between two point clouds. 3DRegNet [22] goes one step further to learn transformation between point clouds end-to-end. Recently, REGTR [38] incorporate self-attention and cross-attention mechanisms and MAC [43] utilizes graph networks to further improve the robustness of end-to-end point cloud registration. GeoTransformer [26] proposes a geometric transformer to match superpoint [9] features and utilizes an overlap-aware circle loss for better convergence. New approaches are constantly being proposed, proving the importance of this task in scene reconstructions.

3D scene representation

Furukawa and Ponce [11] provide a comprehensive classification of 3D reconstruction methods, categorizing them into four primary scene representations: volumetric fields [30, 24], point clouds [29], 3D meshes [15, 39], and depth maps [37, 31, 12]. Except for these representations, NeRF [19] introduces an innovative approach by leveraging a neural implicit field to model the scene. NeRF utilizes an MLP network to optimize a 5D function (3d position plus 2d viewing direction) from a set of training images which can be used to implicitly model the scene. It has shown impressive results in image reconstruction and novel view synthesis and is widely recognized as the first photorealistic 3D scene reconstruction method. Various types of NeRFs have been proposed for acceleration [20, 5, 32] and better rendering quality [3, 2, 34]. Another recent advancement, 3D Gaussian splatting [16] utilizes explicit 3D Gaussians to represent the scene. Each Gaussian is characterized by a covariance matrix, a center point, and opacity for a flexible optimization regime. The model’s efficient differentiable rasterization implementation and well-designed architecture enable rapid training and real-time rendering. Moreover, The optimization strategy is cleverly designed to adaptively control the Gaussians for ensuring very high rendering quality. Despite the fast innovation of scene representations, 3D registration remains to be an important issue for stable large-scale reconstructions, thus developing new registration methods for different representations is crucial.

NeRF Registration

Neural Implicit Field [19] has been widely accepted as a new scene representation, several methods have been proposed to do NeRF registration. NeRF2NeRF [14] utilizes human-annotated key points to obtain an initial transformation and refines it using a surface field distilled from a pre-trained NeRF. DReg-NeRF [6] extracts features from the occupancy grid of NeRF and applies a decoupling model [38] for NeRF registration, eliminating the need for human interaction in the registration process. However, it’s hard to generalize to larger scenes due to its global feature-extracting strategy. NeRFuser [10] directly uses the structure from motion method to estimate the transformation using rendered images from NeRF which is very time-consuming. CL-NeRF [36] concentrates on the continual learning of NeRF models and proposes an expert adaptor for learning newly changed scenes without finetuning the whole network. Most recently, 3D Gaussian splatting has been proposed as a promising scene representation, to the best of our knowledge, we are the first to propose registration methods for 3D Gaussian Splatting and achieve SOTA performance with faster registration speed and better rendering quality. Moreover, continual learning and modifying scenes can be naturally done using our pipeline.

3 Method

In this section, we present our proposed GaussReg for 3D Registration with Gaussian Splatting (GS). The overall architecture is illustrated in Figure 2.

Figure 2: The architecture of GaussReg. Please refer to the text for detailed architecture.

3.1 Overview

As shown in Figure 2, the proposed GaussReg mainly consists of two stages, including the Coarse Registration, and the Image-Guided Fine Registration. Here we give a brief introduction to the entire process. Assuming two overlapping scenes A and B, each with its own GS model, only the camera poses of all training images are saved and accessible. We denote the camera poses of all training images as ${\{C_i^A=(R_i^A,T_i^A)\}}_{i=0}^N$ and ${\{C_j^B=(R_j^B,T_j^B)\}}_{j=0}^M$ for A and B respectively. The GS models are denoted as $Gaussia{n}_A$ and $Gaussia{n}_B$ and the derived point clouds from GS models are termed as $Point{s}_A$ and $Point{s}_B$. Our goal is to discover the rigid transformation $\{s,R,T\}$ that makes scene B align with A, where $s\in ℝ$ means scale factor, $R\in {ℝ}^{3\times 3}$ means rotation matrix, and $T\in {ℝ}^3$ represents translation vector. The coarse registration directly accepts $Point{s}_A$ and $Point{s}_B$ as input, and output a coarse transformation $\{s_c,R_c,T_c\}$. Since the extracted point cloud from a GS model tends to be noisy and distorted, the coarse alignment often needs to be more accurate. Then, in the image-guided fine registration, we first locate a highly overlapping region based on the coarse alignment result. Around the highly overlapping region, two subsets of cameras are then selected from $\{C_i^A\}$ and $\{C_j^B\}$ respectively, from which we render several images. After that, an Image-Guided 3D (I3D) Feature Extraction is adopted to obtain volumetric features from images, which are used for subsequent local matching, ultimately achieving the accurate transformation output $\{s_f,R_f,T_f\}$.

3.2 Coarse Registration

As we all know, the GS model is stored in the form of 3D gaussians. Each 3D gaussian stores the position $(x,y,z)$, opacity $\alpha$, rotation, scale, and the coefficients of the spherical harmonics. First, we select those confident points with the opacity $\alpha$ greater than a threshold (0.7 is chosen empirically). For each sampled point, the color $(r,g,b)$ is determined via spherical harmonic functions. Finally, for every point in $Point{s}_A$ or $Point{s}_B$, we use $(x,y,z,\alpha ,r,g,b)$ as the input channel to feed into the coarse registration pipeline.

As shown in Figure 2, the Coarse Registration follows the workflow as GeoTransformer [26], we extract multi-scale features of each point cloud through a shared KPConv-FPN [33]. The coarsest level point features $F_A^{low}$ and $F_B^{low}$ are used for Superpoint Match and the finest level point features $F_A^{high}$ and $F_B^{high}$ are used for Point Match. The process of Superpoint Match refers to Geotransformer [26]. Noted that in Point Match, we directly utilize the ICP [4] algorithm to obtain the coarse registration results between GS, instead of the Local-to-Global Registration in Geotransformer [26].

Training Strategy and Loss Function

Due to the scale uncertainty in monocular video reconstruction, we performed data augmentation not only on rotation and translation but also on scaling for the input Gaussian point cloud. Even though we normalized the scale of input point clouds within a certain range, such data augmentation still preserves the diversity of relative scale differences between the point clouds to be matched.

We apply two loss functions (overlap-aware circle loss and point matching loss) from the GeoTransformer [26] to constrain our coarse registration network.

Figure 3: The illustration of our overlap image selection and I3D feature extraction.

3.3 Image-Guided Fine Registration

Since the GS model doesn’t impose specific geometric constraints during training, resulting point clouds may exhibit some degree of distortion. Relying solely on GS models might not guarantee accurate registration results. Considering that GS inherently contain detailed image information, an image-guided fine registration is proposed. Our key idea is to first locate overlapping regions between scene $A$ and scene $B$ and render some training images covering the region to support more precise geometric features for fine alignment. Specifically, as shown in Figure 2, our Image-Guided Fine Registration primarily involves two steps: 1) Efficiently and accurately selecting highly overlapping cameras and rendering images accordingly; 2) Utilizing these images to construct volumetric features for further fine registration.

Overlap Image Selection

As shown in Figure 3, the main goal of this part is to find two small subsets of cameras, from $\{C_i^A\}$ and $\{C_j^B\}$ respectively, which share as large common perspective area as possible. Before selection, we first uniformly sampled two subsets $\{C_i^a\}$ and $\{C_j^b\}$ to reduce computational cost, and then apply $\{s_c,R_c,T_c\}$ on $\{C_i^b\}$ for a coarse alignment, causing $\{{\widehat{C}}_i^b\}$. Each subset contains $30$ images in our experiments. Our selection follows 3 steps: 1) For every pair $(C_i^a,{\widehat{C}}_j^b)$, we calculate the cosine value of the angle between their camera orientations. Finally, top-k closest pairs will be kept, where $k=10$ in our experiments. Thanks to the coarse alignment, this step can accurately and quickly removes many useless pairs; 2) To achieve more accurate selection, for each pair $(C_p^a,{\widehat{C}}_q^b)$ remained after step 1, we further calculate the area of their perspective sharing. To do so, two low-resolution depth maps $d_p^a$ and $d_q^b$ are rendered from $Gaussia{n}_A$ and $Gaussia{n}_B$ respectively. Then, we calculate how portion of points derived from $d_p^a$ can be seen from ${\widehat{C}}_q^b$ and how portion of points derived from $d_q^b$ can be seen from $C_p^a$. With the evaluation of the averaged portion, we find the closest pair $(C_{i0}^a,{\widehat{C}}_{j0}^b)$. Thanks to the fast rendering speed of GS, the depth map rendering is done efficiently; 3) We finally pick two subsets of training cameras respectively in the neighborhood of $C_{i0}^a$ and ${\widehat{C}}_{j0}^b$. Under the selected cameras, the image sets $I_A$ and $I_B$ are obtained by rendering from $Gaussia{n}_A$ and $Gaussia{n}_B$ to be fed into the next feature extraction stage.

Image-Guided 3D Feature Extraction

As shown in Figure 3, we adopt the principle of multi-view stereo (MVS) by utilizing images to assist in estimating the depth and extracting volumetric features of the reference image. Without loss of generality, we use scene $A$ as an example in the following description. First of all, we input $I_A$ into a 2D convolutional neural network to get features $Re{f}_A,{\{Sr{c}_A^k\}}_{k=0}^n$, which turn into the cost volume $Cos{t}_A$ according to the depth hypotheses ${\{d_l\}}_{l=0}^D$ by differentiable homography. Building the cost volume requires the minimum and maximum distances, which can be automatically computed from the rendered depth map of the reference image. Followed by the 3DCNN regularization, the probability volume $P_A\in R^{D\times H\times W}$ and feature volume $F_A\in R^{C\times D\times H\times W}$ are obtained from the cost volumes, where $C$ is the number of feature channels, and $(H,W)$ is the resolution of $Re{f}_A$. For any pixel $p$ on $Re{f}_A$, our network predicts a probability distribution ${\{P_A^l(p)\}}_{l=0}^D$. We pick out $l_0$ satisfying:

$$l_0=\underset{l=0,1,\mathrm{\dots },D-1}{\arg \max }\{P_A^l(p)+P_A^{l+1}(p)\},$$ (1)

where $P_A^l(p)$ represents the probability of the pixel $p$ being at depth $d_l$. The feature from cost volume $F_A^{Cost}(p)$, predicted depth $Dept{h}_A(p)$ and confidence map $Con{f}_A(p)$ are calculated by:

$F_A^{Cost}\left(p\right)=F_A^{l_0}\left(p\right)\cdot {\displaystyle \frac{P_A^{l_0}\left(p\right)}{P_A^{l_0}\left(p\right)+P_A^{l_0+1}\left(p\right)}}+F_A^{l_0}\left(p\right)\cdot {\displaystyle \frac{P_A^{l_0+1}\left(p\right)}{P_A^{l_0}\left(p\right)+P_A^{l_0+1}\left(p\right)}},$ (2)

$Dept{h}_A\left(p\right)=d_{l_0}\cdot {\displaystyle \frac{P_A^{l_0}\left(p\right)}{P_A^{l_0}\left(p\right)+P_A^{l_0+1}\left(p\right)}}+d_{l_0+1}\cdot {\displaystyle \frac{P_A^{l_0+1}\left(p\right)}{P_A^{l_0}\left(p\right)+P_A^{l_0+1}\left(p\right)}},$

$Con{f}_A\left(p\right)=P_A^l\left(p\right)+P_A^{l+1}\left(p\right).$

Then we concatenate $Re{f}_A$, $F_A^{Cost}$, and $Dept{h}_A$, and pass them through convolutional layers. After confidence-based filtering, we obtain high-resolution feature $f_A^{high}$ and low-resolution feature $f_A^{low}$. This process can be described as

$f_A^{high}=Conv\left(Concate(Re{f}_A,F_A^{Cost},Dept{h}_A)\right)\left[Con{f}_A>Mean\left(Con{f}_A\right)\right],$ (3)

$f_A^{low}=Conv\left(f_A^{high}\right)\left[Con{f}_A>Mean\left(Con{f}_A\right)\right],$

$f_B^{high}$ and $f_B^{low}$ are obtained in the same manner. Next, we project the features into the coordinate system of $Gaussia{n}_A$ according to the corresponding depth maps. Finally, following the same procedure as in coarse registration, we obtain the fine registration result $\{s_f,R_f,T_f\}$.

Training Strategy and Loss Function

Overlap Image Selection is not involved in the training of the fine registration network. We randomly sample pairs of multi-view images with overlap from ScanNet dataset for training. During training, we also apply data augmentation to the camera extrinsic.

Our loss function mainly consists of two parts, depth loss and registration loss. Depth loss is a cross-entropy loss to supervise the probability volume:

$$L_{depth}=\sum _{p\in {\mathrm{\Omega }}_A}-P_A^{gt}(p)log{P}_A(p)+\sum _{p\in {\mathrm{\Omega }}_B}-P_B^{gt}(p)log{P}_B(p),$$ (4)

where ${\mathrm{\Omega }}_A$ and ${\mathrm{\Omega }}_B$ are the sets of valid points. $P_A^{gt}(p)$ and $P_B^{gt}(p)$ denote the one-hot labels from the ground-truth depth of $p$. $P_A(p)$ and $P_B(p)$ denote the predicted probability distribution of $p$. Registration loss $L_{regis}$ is the same as loss function used in coarse registration. Therefore, our total loss in the fine registration network is:

$$L_{total}=\lambda L_{depth}+L_{regis},$$ (5)

where $\lambda =10$ in our experiments.

Figure 4: Visualization of our final registration results on ScanNet-GSReg and GSReg. The first two columns are visualizations of GS point clouds to be registered. The last two columns are visualizations of our final registration and ground-truth results.

3.4 Gaussian Splatting Fusion and Filtering

After obtaining the final registration result, it is time to merge the two GS models. To transform $Gaussia{n}_B$ into the coordinate system of $Gaussia{n}_A$, denoted as $Gaussia{n}_{B\to A}$, we start by transforming the position of the 3D gaussian:

$${(x_{B\to A},y_{B\to A},z_{B\to A})}^T=s_f{R}_f{(x_B,y_B,z_B)}^T+T_f.$$ (6)

The opacity is invariant to the transformation ${\alpha }_{B\to A}={\alpha }_B$. The rotation $R_{B\to A}\in R^{3\times 3}$ and scale $S_{B\to A}\in R^3$ of the 3D gaussian can be computed as:

$R_{B\to A}=R_f{R}_B,$ (7)

$S_{B\to A}=s_f{S}_B.$

From the properties of spherical harmonics (SH) coefficients, we know that the rotation of SH coefficients is a linear transformation of the SH coefficients, and the rotation of each order of SH coefficients can be performed separately. Hence, for the $i$-th order of SH coefficients, we can obtain the transformation of SH coefficients through the following approach: 1) Select any 2i+1 unit vectors $u_0,\mathrm{\dots },u_{2i+1}$, let $Q=(SH(u_0),\mathrm{\dots },SH(u_{2i+1}))$, where $SH$ is the function that projects the direction vector to the corresponding SH values; 2) Apply transformation $\{s_f,R_f,T_f\}$ to vectors $u_0,\mathrm{\dots },u_{2i+1}$ to yield $\widehat{u_0},\mathrm{\dots },\widehat{u_{2i+1}}$; 3) $(SH(\widehat{u_0}),\mathrm{\dots },SH(\widehat{u_{2i+1}}))Q^{-1}$ is the transformation matrix of SH coefficients. Note that, it is difficult to choose vectors $u_0,\mathrm{\dots },u_{2i+1}$ to ensure that $Q$ is invertible, so in our experiments, we use the pseudo inverse as an approximation to inverse while calculating $Q^{-1}$. Finally, we merge the 3D gaussians in $Gaussia{n}_A$ closer to the center of $A$ with the 3D gaussians in $Gaussia{n}_B$ closer to the center of $B$ to get $Gaussia{n}_{A+B}$.

4 Experiment

4.1 Experiment Setup

Table 1: Evaluation on the ScanNet-GSReg dataset. $\downarrow$ means lower is better, and $\uparrow$ means higher is better. We believe that data preprocessing is also part of the method, so we include the time of obtaining point cloud from GS in the time reported. Here HLoc [27]* means HLoc [27] (SuperPoint [9] + SuperGlue [28]).

Methods	RRE$\downarrow$	RTE$\downarrow$	RSE$\downarrow$	Succss Ratio$\uparrow$	Time(s)$\downarrow$
HLoc [27]*	2.725	0.099	0.098	0.756	212.3
FGR [44]	157.126	3.328	0.268	1.000	3.4
REGTR [38]	80.095	2.768	0.408	1.000	3.5
Ours	2.827	0.042	0.032	1.000	4.8

Dataset

As there is currently no scene-level dataset available for our task, it is necessary for us to create a dataset in order to evaluate GS registration. ScanNet [7] is a frequently used 3D dataset for indoor scenes, consisting of $1513$ training scenes and $100$ test scenes. Each scene in ScanNet includes camera intrinsics, a sequence of images, along with the corresponding camera extrinsics and depth maps. Therefore, we decide to build a dataset based on ScanNet, called ScanNet-GSReg dataset. First, we randomly sample two continue image sequences from each scene. Each sequence contains $80$ to $120$ images, and the sampling interval ranges from $1$ to $5$. The overlap ratio, calculated as the proportion of repeated images between two sequences, ranges from $0.2$ to $0.8$. Then, we apply random transformations to each set of camera extrinsics independently to simulate the inconsistency between the world coordinates of the two sequences and record these two transformations as the ground-truth transformation between their world coordinates. Using these image sequences and corresponding camera parameters, we reconstruct the GS models separately. Each model undergoes $10000$ iterations of training. Eventually, after excluding cases of failed initial point cloud generation or unsuccessful GS reconstruction, we obtain $1297$ training samples and $82$ test samples. Furthermore, to validate the generalization of our method, we collected 10 real-world scenes for testing, called GSReg dataset, which includes $6$ indoor and $4$ outdoor scenes. For each scene, we record two videos. First, we use HLoc [27] (SuperPoint [9] as the feature extractor and SuperGlue [28] as the matcher) to obtain the camera poses individually for each video, and then combine the two videos for a joint camera pose estimation to obtain the ground-truth transformation between the two GS models. To evaluate the performance of GaussReg on objects, we also conduct tests on the Objaverse dataset [8] used in DReg-NeRF [6], whose test set contains 44 objects.

Metric

We refer to metrics of point cloud registration as in [26] and modify them to account for scale factors. Finally, we evaluate GaussReg on the ScanNet-GSReg and GSReg datasets with three metrics: 1) Relative Rotational Error (RRE), the geodesic distance between the estimated and ground-truth rotation matrix; 2) Relative Translation Error (RTE), the ratio of the Euclidean distance between the estimated and ground-truth translation vectors to the norm of the ground-truth translation vector; 3) Relative Scale Error (RSE), the ratio of the Euclidean distance between the estimated and ground-truth scale factors to the ground-truth scale factor. For a fair comparison, we follow DReg-NeRF [6] to evaluate GaussReg on the Objaverse dataset with two metrics: 1) Relative Rotational Error (RRE); 2) Absolute Translational Error (ATE), the Euclidean distance between the estimated and ground-truth translation vectors.

Implementation Details

Our GaussReg is merely trained on the ScanNet-GSReg training set and evaluated on the ScanNet-GSReg test set, Objaverse test set, and GSReg dataset. Our method was implemented with PyTorch [25]. In the coarse registration network, we limit the number of input points to $30000$ during training. In the image-guided fine registration network, we render $n=5$ images per GS model as input and set the number of depth hypotheses to $D=64$. Both networks are trained separately for $40$ epochs with a batch size of $1$. The learning rate starts from $1e-4$ and decays exponentially by $0.05$ every epoch.

Table 2: Evaluation on the Objaverse dataset. $\downarrow$ means lower is better.

Methods	RRE$\downarrow$	ATE$\downarrow$
FGR [44]	61.59	13.50
REGTR [38]	113.78	43.31
Dreg-NeRF [6]	9.67	3.85
Ours w/o. fine	2.47	3.46

Table 3: Evaluation on the GSReg dataset. $\downarrow$ means lower is better.

Methods	RRE$\downarrow$	RTE$\downarrow$	RSE$\downarrow$
Ours w/o. fine	6.904	0.074	0.051
Ours	2.989	0.065	0.047

4.2 Comparison with Other Methods

Evaluation on the ScanNet-GSReg Dataset

Due to the maturity of Structure from Motion (SFM) technology, a natural approach for 3D registration with GS is to render a large number of images and utilize SFM for joint registration. Therefore, we select the current SOTA method, HLoc [27] (SuperPoint [9] + SuperGlue [28]), as the baseline for comparison on ScanNet. In the subsequent discussion, we refer to HLoc [27] (SuperPoint [9] + SuperGlue [28]) as HLoc for brevity. For the two GS models to be registered, we uniformly sample $30$ training poses each to render images, and use $60$ images in total for HLoc to estimate pose. We can obtain the registration result of the two GS models following the procedure described in NeRFuser [10]. We also evaluate traditional point cloud registration method Fast Global Registration (FGR) [44] and deep point cloud registration method REGTR [38] (retrained on 3DMatch) by inputting the point cloud from GS. FGR and REGTR are also followed by the ICP solver with scaling to output the transformation results, and we also limit the number of input points to $30000$. The quantitative results are shown in Table 1, where the Success Ratio indicates the portion of successful registrations. As shown in Table 1, for $82$ scenes in ScanNet-GSReg, HLoc only registers $75.6\%$ of them successfully, while our method achieves a $100\%$ success ratio. For indoor scenes in ScanNet-GSReg, SuperPoint [9] sometimes fails to extract effective keypoints, leading to registration failures. Our method outperforms HLoc in RTE and RSE metrics and is comparable in RRE. Notably, our method was significantly faster than HLoc ($4.8s$ vs. $212.3s$). FGR and REGTR are slightly faster than our GaussReg, however, they perform much worse than ours. We think the reason is that the point cloud from GS is much noisier than scanning data. Visualizations of our method on the ScanNet-GSReg test set are presented in the first two rows of Figure 4. More visual results can be found in Supplementary Material. These experiments fully demonstrate the efficiency and accuracy of our method.

Evaluation on the Objaverse Dataset

For a fair comparison on the Objaverse dataset [8] used in DReg-NeRF, we assume there is no scale difference between the two GS models as in DReg-NeRF [6]. In addition, we do not adopt training poses, and only use our proposed coarse registration for comparison. As shown in Table 3, our coarse registration method (ours w/o. fine) significantly outperforms other methods without fine-tuning, demonstrating its strong generalization capability to objects.

Evaluation on the GSReg Dataset

The ground-truth registration results of our GSReg dataset are obtained when HLoc was successful. As shown in Table 3, our method achieves registration results close to HLoc without fine-tuning, proving the strong generalizability of our approach. Moreover, our method (ours) significantly outperforms our coarse registration (ours w./o. fine), proving the effectiveness of our fine registration. Visualizations of our method on the GSReg dataset are presented in the last two rows of Figure 4.

Table 4: Ablation study of image-guided fine registration on the ScanNet-GSReg dataset. $\downarrow$ means lower is better, and $\uparrow$ means lower is better.

Index	Methods	RRE$\downarrow$	RTE$\downarrow$	RSE$\downarrow$	Succss Ratio$\uparrow$	Time(s)$\downarrow$
1	Hloc [27]	2.725	0.099	0.098	0.756	212.3
2	Ours w./o. fine	3.403	0.061	0.034	1.000	3.7
3	Ours w./o. fine + HLoc	1.104	0.186	0.278	0.512	206.8
4	Ours	2.827	0.042	0.032	1.000	4.8

4.3 Ablation Study

To deeply analyze GaussReg, we conduct detailed ablation studies on the ScanNet-GSReg dataset to evaluate the effectiveness of the proposed components.

Effectiveness of Image-Guided Fine Registration

HLoc can also utilize image information to refine the coarse registration. Therefore, to validate the effectiveness of image-guided fine registration, we directly combine coarse registration with HLoc. After obtaining the coarse registration result, we use overlap image selection to select two sets of multi-view images, $Image{s}_A$ and $Image{s}_B$, and jointly use $Image{s}_A$ and $Image{s}_B$ for pose estimation with HLoc. As shown in Table 4, by comparing Index-2 with Index-4, we can see that the performance is improved, which demonstrates the effectiveness of our image-guided fine registration. Comparing Index-2 and Index-3, we find that although HLoc shows lower RRE, its success ratio is very low ($51.2\%$), whereas our fine registration not only outperforms HLoc in RTE and RSE metrics but also has a higher success ratio ($100\%$). Meanwhile, our fine registration is faster than HLoc ($4.8s$ vs. $206.8s$). In addition, we explore the effect of the top-k pairs of cameras we kept in overlap image selection. Hence, we vary k from 5 to 30. In Table 6, there is almost no change in performance when k is larger than 10 and the performance drops when k is smaller than 10. For the sake of accuracy and efficiency, we believe that 10 is enough for k.

Table 5: Ablation study with different k in overlap image selection on ScanNet-GSReg. $\downarrow$ means lower is better.

Top-k	RRE$\downarrow$	RTE$\downarrow$	RSE$\downarrow$
5	3.677	0.115	0.079
10	2.827	0.042	0.032
20	2.604	0.063	0.044
30	2.311	0.091	0.028

Table 6: Ablation study of image-guided 3D feature extraction on ScanNet-Reg. $\downarrow$ means lower is better.

Index	Method	RRE$\downarrow$	RTE$\downarrow$	RSE$\downarrow$	RDE$\downarrow$
5	Ours w/o. I3D	3.169	0.036	0.061	0.066
6	Ours	2.827	0.042	0.032	0.080

Effectiveness of Image-Guided 3D Feature Extraction

Here, we also report the Relative Depth Error (RDE), which is the ratio of the Euclidean distance between the estimated and ground-truth depth to the ground-truth depth. As shown in Table 6, in Index-5, we remove the image-guided 3D (I3D) feature extraction. Instead, we use MVSNet [12] to calculate depth and project depth maps to obtain two point clouds, which serve as input to KPConv-FPN [33] to extract features for registration refinement. Comparing Index-5 and Index-6, we observe that although Index-5 has better depth estimation accuracy, the registration results are poor, proving that extracting geometric information from images complements feature descriptors extraction.

4.4 Results of Gaussian Splatting Fusion and Filtering

In Figure 5, we present some quantitative results on GSReg dataset to demonstrate the effectiveness of our GS fusion and filtering. Please refer to the video attachment in Supplementary Material for the dynamic presentation. Our GS fusion and filtering strategy successfully merges the two GS models.

5 Discussion

Figure 5: Quantitative Results on the GSReg dataset. The first two rows are indoor scenes, and the last two rows are outdoor scenes. The first and third columns are rendering images from GS [16] models of Scene A and Scene B. The second and last columns are rendering images from our fused GS model.

Limitations and Future Work

We only adopt a simple strategy to fuse and filter two GS models. For some more complex situations, the fusion in our way is imperfect. For instance, when two scenes are captured at different times, changes in lighting can result in differing appearances for two scenes. Consequently, the fused GS model obtained through our strategy may exhibit inconsistencies at the fusion boundary. Future work can further explore to address this issue.

Conclusion

The advent of Neural Radiance Fields (NeRF) has transformed the landscape of 3D scene representation, necessitating advancements in registration methodologies. However, the registration of NeRF representations for large-scale scenes remains underexplored due to the inherent complexities of implicitly modeled geometric relationships. The recent introduction of Gaussian Splatting (GS) has significantly enhanced NeRF by introducing explicit 3D gaussians, facilitating rapid rendering while maintaining high quality. In this study, we introduce GaussReg, a pioneering coarse-to-fine framework that utilizes GS for 3D registration with GS. The coarse phase leverages existing point cloud registration methods to establish a preliminary alignment for input GS point clouds. We innovatively devise an image-guided fine registration strategy that incorporates rendered images from these Gaussian points, enriching geometric details for accurate alignment. To comprehensively evaluate our approach, we construct a benchmark consisting of scenes from ScanNet and several in-the-wild scenes. Our experimental results show GaussReg’s state-of-the-art performance across multiple datasets.