instantsplat

packaged InstantSplat

This repo is the refactored python training and inference code for InstantSplat. Forked from commit 2c5006d41894d06464da53d5495300860f432872. We refactored the original code following the standard Python package structure, while keeping the algorithms used in the code identical to the original version.

Initialization methods:

• DUST3R (same method used in InstantSplat)
• MAST3R (same method used in Splatt3R)
• COLMAP Sparse reconstruct (same method used in gaussian-splatting)
• COLMAP Dense reconstruct (use patch_match_stereo, stereo_fusion, poisson_mesher and delaunay_mesher in COLMAP to reconstruct dense point cloud for initialization)

Prerequisites

• Pytorch (v2.4 or higher recommended)
• CUDA Toolkit (12.4 recommended, should match with PyTorch version)

Install (PyPI)

pip install --upgrade instantsplat

Install (Build from source)

pip install --upgrade git+https://github.com/yindaheng98/InstantSplat.git@main

If you have trouble with gaussian-splatting, you can install it from source:

pip install --upgrade git+https://github.com/yindaheng98/gaussian-splatting.git@master

Install (Development)

Install gaussian-splatting. You can download the wheel from PyPI:

pip install --upgrade gaussian-splatting

Alternatively, install the latest version from the source:

pip install --upgrade git+https://github.com/yindaheng98/gaussian-splatting.git@master

git clone --recursive https://github.com/yindaheng98/InstantSplat
cd InstantSplat
pip install tqdm plyfile scikit-learn numpy tifffile triton xformers
pip install --target . --upgrade --no-deps .

(Optional) Install xformers and triton for faster depth anything:

pip install triton xformers

(Optional) If you prefer not to install gaussian-splatting in your environment, you can install it in your InstantSplat directory:

pip install --target . --no-deps --upgrade git+https://github.com/yindaheng98/gaussian-splatting.git@master

Download model

wget -P checkpoints/ https://download.europe.naverlabs.com/ComputerVision/DUSt3R/DUSt3R_ViTLarge_BaseDecoder_224_linear.pth
wget -P checkpoints/ https://download.europe.naverlabs.com/ComputerVision/DUSt3R/DUSt3R_ViTLarge_BaseDecoder_512_linear.pth
wget -P checkpoints/ https://download.europe.naverlabs.com/ComputerVision/DUSt3R/DUSt3R_ViTLarge_BaseDecoder_512_dpt.pth
wget -P checkpoints/ https://download.europe.naverlabs.com/ComputerVision/MASt3R/MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric.pth
wget -P checkpoints/ https://huggingface.co/depth-anything/Depth-Anything-V2-Small/resolve/main/depth_anything_v2_vits.pth
wget -P checkpoints/ https://huggingface.co/depth-anything/Depth-Anything-V2-Base/resolve/main/depth_anything_v2_vitb.pth
wget -P checkpoints/ https://huggingface.co/depth-anything/Depth-Anything-V2-Large/resolve/main/depth_anything_v2_vitl.pth

Running

• Initialize coarse point cloud and jointly train 3DGS & cameras

# Option 1: init and train in one command
python -m instantsplat.train -s data/sora/santorini/3_views -d output/sora/santorini/3_views -i 1000 --init dust3r
# Option 2: init and train in two separate commands
python -m instantsplat.train -i dust3r -d data/sora/santorini/3_views -i dust3r # init coarse point and save as a Colmap workspace at data/sora/santorini/3_views
python -m instantsplat.train -s data/sora/santorini/3_views -d output/sora/santorini/3_views -i 1000 # train

• Render it

python -m gaussian_splatting.render -s data/sora/santorini/3_views -d output/sora/santorini/3_views -i 1000 --load_camera output/sora/santorini/3_views/cameras.json

See .vscode\launch.json for more command examples.

Usage

See instantsplat.initialize, instantsplat.train and gaussian_splatting.render for full example.

Also check yindaheng98/gaussian-splatting for more detail of training process.

Gaussian models

Use CameraTrainableGaussianModel in yindaheng98/gaussian-splatting

Dataset

Use TrainableCameraDataset in yindaheng98/gaussian-splatting

Initialize coarse point cloud and cameras

from instant_splat.initializer import Dust3rInitializer
image_path_list = [os.path.join(image_folder, file) for file in sorted(os.listdir(image_folder))]
initializer = Dust3rInitializer(...).to(args.device) # see instant_splat/initializer/dust3r/dust3r.py for full options
initialized_point_cloud, initialized_cameras = initializer(image_path_list=image_path_list)

Create camera dataset from initialized cameras:

from instant_splat.initializer import TrainableInitializedCameraDataset
dataset = TrainableInitializedCameraDataset(initialized_cameras).to(device)

Initialize 3DGS from initialized coarse point cloud:

gaussians.create_from_pcd(initialized_point_cloud.points, initialized_point_cloud.colors)

Training

Trainer jointly optimize the 3DGS parameters and cameras, without densification

from instant_splat.trainer import Trainer
trainer = Trainer(
    gaussians,
    scene_extent=dataset.scene_extent(),
    dataset=dataset,
    ... # see instant_splat/trainer/trainer.py for full options
)

InstantSplat: Sparse-view SfM-free Gaussian Splatting in Seconds

This repository is the official implementation of InstantSplat, an sparse-view, SfM-free framework for large-scale scene reconstruction method using Gaussian Splatting. InstantSplat supports 3D-GS, 2D-GS, and Mip-Splatting.

Free-view Rendering

https://github.com/zhiwenfan/zhiwenfan.github.io/assets/34684115/748ae0de-8186-477a-bab3-3bed80362ad7

TODO List

• Confidence-aware Point Cloud Downsampling
• Support 2D-GS
• Support Mip-Splatting

Acknowledgement

This work is built on many amazing research works and open-source projects, thanks a lot to all the authors for sharing!

• Gaussian-Splatting and diff-gaussian-rasterization
• DUSt3R

Citation

If you find our work useful in your research, please consider giving a star :star: and citing the following paper :pencil:.

@misc{fan2024instantsplat,
        title={InstantSplat: Unbounded Sparse-view Pose-free Gaussian Splatting in 40 Seconds},
        author={Zhiwen Fan and Wenyan Cong and Kairun Wen and Kevin Wang and Jian Zhang and Xinghao Ding and Danfei Xu and Boris Ivanovic and Marco Pavone and Georgios Pavlakos and Zhangyang Wang and Yue Wang},
        year={2024},
        eprint={2403.20309},
        archivePrefix={arXiv},
        primaryClass={cs.CV}
      }