Render 3D Gaussian Splats using Metal on Apple platforms (iOS/iPhone/iPad, macOS, and visionOS/Vision Pro)
This is a Swift/Metal library for rendering scenes captured via the techniques described in 3D Gaussian Splatting for Real-Time Radiance Field Rendering. It will let you load up a PLY and visualize it on iOS, macOS, and visionOS (using amplification for rendering in stereo on Vision Pro). Modules include
- MetalSplatter, the core library to render a frame
- PLYIO, for reading and writing binary or ASCII PLY files; this is standalone, feel free to use it if you just have a hankering to load up some PLY files for some reason.
- SplatIO, a thin layer on top of PLYIO to interpret these PLY files (as well as .splat files) as sets of splats
- SampleApp, a mini app to demonstrate how to use the above (based on Apple template code) -- don't expect much, it's intentionally minimal, just an illustration
- SampleBoxRenderer, a drop-in replacement for MetalSplatter for debugging integration, which just renders the cube from Apple Metal template
You're right, the documentation is entirely missing; it's a major TODO list item. In the meantime, feel free to try out the sample app -- just, like I said, don't expect much.
- Get yourself a gaussian splat PLY (or .splat) file (see Resources below)
- Clone the repo and open SampleApp/MetalSplatter_SampleApp.xcodeproj
- If you want to run on iOS/visionOS, select your target and set your development team and bundle ID in Signing & Capabilities. On macOS, just have at it.
- Make sure your scheme is set to Release mode. Loading large files in Debug is more than an order of magnitude slower.
- Run
- Note: framerate will be better if you run without the debugger attached (hit Stop in Xcode, and go run from the app from the Home screen)
-
The MetalSplatter viewer is a simple, official Vision Pro app based on this library. This is different from the minimal included sample app (for instance, it has camera controls and a splat gallery). Confusingly, both the (open source) library and (non-open-source) app are called MetalSplatter, and both are by scier.
-
OverSoul for Vision Pro: "Capture, share, and interact with spatial photos, 3D models and immersive spaces in a vibrant social ecosystem designed for the next generation of spatial computing"
-
Know of another project using MetalSplatter? Let us know!
- Looking for .splat files? Here are a few suggestions:
- Capture your own by using a camera or drone, then use Nerfstudio to train the splat
- Capture one with Luma AI's iPhone app, and export it in "splat" format
- Use the scene data from the original paper
- RadianceFields.com is a great source to track news and articles about 3DGS, NeRFs, and related technology and tools (for instance news about MetalSplatter), and the community surrounding it
- MrNeRF's Awesome 3D Gaussian Splatting Resources is exactly what it says on the label - in particular, an exhaustive and frequently-updated list of 3DGS-related research
- There are a variety of other renderer implementations available with source, such as
- Kevin Kwok's WebGL implementation and online demo
- Mark Kellogg's three.js implementation and online demo
- Aras Pranckevičius's Unity implementation and series of very insightful blog posts: 1, 2, 3
- The original paper's reference implementation
