(Experimental) Gaussian splatting + WebGL

[brentyi]

I spent some time studying + adapting https://github.com/antimatter15/splat/. It's really cool stuff.

splat.mp4

Work in progress: there's a lot of low hanging fruit for optimization, and the current implementation clashes with the <EffectComposer /> we use for outline/hover effects.

[brgrp]

Hey, I just tried to run the demo with Chrome and Firefox, but my display is getting incredibly slow. GPU and CPU usage stays low.

Ubuntu 22.04 LTS + GTX3080
Chrome: 117.0.5938.132 (64-bit)
Firefox: 117.0.1 (64-bit)
NVIDIA-SMI 525.125.06 Driver Version: 525.125.06

Does anyone have an idea what the problem could be?

[jkulhanek]

Can you confirm hardware acceleration is enabled (chrome://gpu)?

[brgrp]

Can you confirm hardware acceleration is enabled (chrome://gpu)?

Graphics Feature Status

• Canvas: Hardware accelerated
• Canvas out-of-process rasterization: Enabled
• Direct Rendering Display Compositor: Disabled
• Compositing: Hardware accelerated
• Multiple Raster Threads: Enabled
• OpenGL: Enabled
• Rasterization: Hardware accelerated
• Raw Draw: Disabled
• Skia Graphite: Disabled
• Video Decode: Hardware accelerated
• Video Encode: Software only. Hardware acceleration disabled
• Vulkan: Disabled
• WebGL: Hardware accelerated
• WebGL2: Hardware accelerated
• WebGPU: Hardware accelerated

[brentyi]

Which splat file are you loading? Splats are currently rendering smoothly in Chrome on both my laptop (M1 Mac) and phone (Samsung S21+).

[brentyi]

sort timing notes:

# Gaussians	270K	1M
Array.sort	79ms	380ms
JS counting sort	25ms	140ms
C++ counting sort	14ms	53ms

[jkulhanek]

What is the current state of things? Is it ready to be merged soon?

[brentyi]

I'm leaning toward closing the PR; it's working, but:

• Still clashes with outline effects / EffectComposer.
• Doesn't sort properly when there are multiple splat files.
• Can be optimized a lot. (in particular I would want to move more data into textures, as opposed to attributes)

After I implemented this splatting capabilities have also popped up in external libraries (example: https://github.com/pmndrs/drei#Splat); it may make sense to adapt one of these instead.

[a1raman]

Any progress?

[brentyi]

This is merged and the majority of kinks are worked out, but it'll be a while before it hits the stable API.

If you want to try it you can install viser from source and call server.scene._add_gaussian_splats() (or run the example in the examples/experimental directory):

viser/src/viser/_scene_api.py

Lines 935 to 963 in ba17c7c

	def _add_gaussian_splats(
	self,
	name: str,
	centers: onp.ndarray,
	covariances: onp.ndarray,
	rgbs: onp.ndarray,
	opacities: onp.ndarray,
	wxyz: Tuple[float, float, float, float] | onp.ndarray = (1.0, 0.0, 0.0, 0.0),
	position: Tuple[float, float, float] | onp.ndarray = (0.0, 0.0, 0.0),
	visible: bool = True,
	) -> GaussianSplatHandle:
	"""Add a model to render using Gaussian Splatting.
	**Work-in-progress.** This feature is experimental and still under
	development. It may be changed or removed.
	Arguments:
	name: Scene node name.
	centers: Centers of Gaussians. (N, 3).
	covariances: Second moment for each Gaussian. (N, 3, 3).
	rgbs: Color for each Gaussian. (N, 3).
	opacities: Opacity for each Gaussian. (N, 1).
	wxyz: R_parent_local transformation.
	position: t_parent_local transformation.
	visibile: Initial visibility of scene node.
	Returns:
	Scene node handle.
	"""

[fantasy-fish]

I ran the command python examples/experimental/gaussian_splats.py --splat-paths ~/data/test1-splat/point_cloud/iteration_30000/point_cloud.ply. It successfully loaded the ply file with the message "PLY file with num_gaussians=410549 loaded in 0.6525325775146484 seconds". But when I opened http://0.0.0.0:8080, it shows a blank web page in my browser.