Streaming RGB-D Splat Primitives in Unity

A comparison of rendering primitives for a live RGB-D point cloud in Unity: plain points, flat quads, camera-facing billboards, and surface-oriented 2D Gaussian surfels. Each depth pixel is turned into a primitive every frame, straight from the sensor data, with no training, no network, and no per-scene optimisation, then compared on visual completeness and render cost.

It is an extension of the XR Teleoperation project’s perception system, which streamed the point cloud as the naive image-plane baseline and went sparse at close range and oblique angles. After that subject I wanted to try Gaussian splatting, so I picked the perception work back up and built a surfel renderer to orient a small disk to the local surface and keep it continuous. As more of the literature came into view it turned comparative: how much does the choice of primitive actually matter for live RGB-D, and is a surface-oriented surfel worth its cost over simpler options like a camera-facing billboard?

Pipeline

Three GPU-side stages, no per-frame CPU readback:

• Ingest - depth and colour frames uploaded to GPU textures with camera intrinsics
• Depth to surfels (compute shader) - per-pixel Kalman filtering of depth, back-projection to 3D, normals via finite differences, edge detection to flatten discontinuities to billboards, and per-axis tangent/anisotropy for disk stretch and orientation
• Rasterise (vertex + fragment shader) - each point expanded to a quad, oriented and stretched into an ellipse with a Gaussian falloff, rendered opaque with depth-buffer overlap resolution and MSAA on edges

Results

A pre-planned route was recorded and replayed for each stream to keep the viewing path consistent between runs. All settings were held constant; only the feature under test was enabled or disabled. Object detection was CPU-bound throughout (no GPU/OpenVINO), so that overhead is included in its figures.

The flat quad foreshortens at oblique angles and leaves visible gaps up close and side-on. Both the surfels and the billboard remove this. The surfels run at roughly 30 to 45 render fps; the billboard is comparable (about 30 to 40) and holds up visually while being far simpler. The flat quad is faster (40 to 60) only by doing and showing less.

For raw coverage the two are close, but the surface orientation does show up as better object shape: the surfels hold form and occlude correctly as the view moves off-axis, where the billboard faces the viewer and reads more like a flat card. That shape edge costs extra compute and adds some temporal instability.

Billboard vs 2D Gaussian (Kalman)

The headline comparison, both running with the Kalman filter. The billboard fills the surface about as well as the surfels while staying far simpler; the surfels keep better object shape off-axis.

Camera-facing billboard

2D Gaussian surfels

Flat image-plane quad

The naive baseline, drawn first with no temporal smoothing, then with the Kalman filter, then with object detection.

2D Gaussian surfels

The surfels again, without the filter and with object detection added.

Settings

The inspector exposes each stage so a single feature can be toggled while the rest are held constant.

Conclusion

Both primitives clearly beat the flat quad. For just filling the surface, the camera-facing billboard does about as well as the 2D Gaussian surfel, is simpler, and doesn’t flicker like the tilted disks; drawn opaque instead of transparent it would also be faster. The surfel still wins on shape: disks sit on the actual surface, so objects keep their form and occlude correctly as the view moves off-axis.

Rough rule of thumb:

• Billboard for a complete, stable, cheap point cloud viewed roughly head-on, on tight hardware
• 2D Gaussian surfel when orbiting objects and shape or occlusion off-axis matters, and the extra compute is affordable

Repositories

• RGB-D Splat Primitives In Unity