To reproduce, create a webXR scene with a gaussian splat mesh in it. Open it in an Oculus Quest 2 and lean your ear to your shoulder.
This will create the effect of the splats rotating along with the roll of your head, becoming completely misaligned with how they should be viewed. I’ve been trying to fix this behavior with a custom shader but frankly it’s a bit beyond me at the moment! Would love some help.
Sorry, can’t reproduce that. I have tested GS in XR before, so I had to try it myself again. Is this playground not working corectly?
@RaananW
Video (Quest 2): SplatRotation.mp4 - Google Drive
Project files: Splattingfix.zip - Google Drive
Server command: http-server dist -S -C ./server.crt -K ./server.key
Scripts:
Index.ts:
import * as BABYLON from ‘@babylonjs/core’;
import { GaussianSplat } from ‘./GaussianSplat’;
// Get the canvas element from the DOM
const canvas = document.getElementById(‘renderCanvas’) as HTMLCanvasElement;
// Create the Babylon.js engine
const engine = new BABYLON.Engine(canvas, true);
// Create a new scene
const createScene = async () => {
const scene = new BABYLON.Scene(engine);
// Create a camera and attach it to the canvas
const camera = new BABYLON.ArcRotateCamera(‘camera’, -Math.PI / 2, Math.PI / 2.5, 3, new BABYLON.Vector3(0, 0, 0), scene);
camera.attachControl(canvas, true);
// Create a light source
const light = new BABYLON.HemisphericLight(‘light’, new BABYLON.Vector3(0, 1, 0), scene);
// Create a basic sphere
const sphere = BABYLON.MeshBuilder.CreateSphere(‘sphere’, { diameter: 1 }, scene);
// Instantiate the GaussianSplat class
new GaussianSplat(scene);
// Create a plane as the ground
const ground = BABYLON.MeshBuilder.CreateGround(‘ground’, { width: 10, height: 10 }, scene);
// Initialize XR support
const xr = await scene.createDefaultXRExperienceAsync({
floorMeshes: [ground], // Use the ground plane as the floor mesh
});
return scene;
};
// Call the createScene function
createScene().then(scene => {
// Register a render loop to repeatedly render the scene
engine.runRenderLoop(() => {
scene.render();
});
// Watch for browser/canvas resize events
window.addEventListener(‘resize’, () => {
engine.resize();
});
});
GaussianSplat.ts:
import * as BABYLON from ‘@babylonjs/core’;
export class GaussianSplat {
constructor(scene: BABYLON.Scene) {
// Gaussian Splatting
var gs = new BABYLON.GaussianSplattingMesh("Halo", null, scene);
gs.loadFileAsync("https://assets.babylonjs.com/splats/gs_Skull.splat").then(()=>{
gs.position.y = 1.7;
});
}
}
That doesn’t look like a GS issue, it looks like everything else in the scene is rotating incorrectly. What version of babylonjs are you using here? Also, is the demo i shared with you working correctly?
@RaananW same issue there. Not sure what you mean everything in the scene is rotating incorrectly. I am tilting my head to the left and right (touching my ear to my shoulder). The background plane in the scene rotates fine. But the splats are rotating with my head, when they shouldn’t be. I’m using Babylon 7.
Unity even had this problem with their particles and had to implement a fix:
There is a ground object under the GS object. It is rotating with the GS object. Not independently. Unless I a missing something, they are not attached to one another in any way (parenting etc’), so if it was just a GS issue, the ground would have rotated correctly.
Now - what version of babylon are you using? And what version of browser are you using?
I am using Chrome/testing with Meta Quest 2’s default browser and I’m using Babylon 7.
You are misunderstanding; the problem is not that the entire splat mesh object is rotating; the problem is that the individual splats are rotating with my head. Look closely at where I point the controller, at the long grey streaks at the bottom of the object.
Sorry, got it now - This is something for @Cedric to comment on. I’ll check with him.
Hi @Elevons
I’m trying to understand the issue. In this PG, I rotate the camera:
Splats also rotate as I would except them to. Can you see your issue?
Hi @Cedric - you have to rotate your head in VR. Touch your ear to your shoulder and you will see the splats roll with your head.
When it should stay vertical.
I have no VR headset. Can you please make a video? do you repro @RaananW ?
Here you go @Cedric 
I tried editing the shader code, but I can’t get past the “this transform4x4 is not part of the current program” error I’m getting. That being said, I think all you have to do is remove the roll calculation from the billboarding effect that you are applying to the gaussian splat particles.
It looks more like rolling angle is applied 2 times instead of 1.
Can it be something related to perspective matrix update?
If it happens with VR, it should be reproductible with a non-vr setup.
Logically to me if splats are typically non-square objects then they should really never have their roll values updated, since if you roll a rectangle it will always break up the silhouette. But that’s me thinking in typical particle terms, I don’t know if it applies to this technology.
Unity had the same issue, only with VR headsets:
Here were some soloutions that were implemented there until the official fix was implemented (which just disables particle roll IIRC).
Happy to test this with you back and forth if you’d like! My Discord handle is elevons.
I want to think it is not VR related, but multi-viewport related. Might be an issue with the way the viewport/view matrix is applied to the shader?
I can reproduce that in VR, haven’t tried testing that in non VR scenarios
If @Cedric thinks that it might be the rolling angle is applied 2x instead of 1 then it would make sense that we might be applying the perspective matrix twice to each billboard, based on each camera.
I don’t have a VR headset so, hard to repro 
Yes, looks like an issue with view/viewport matrix or view ratio issue.
it might be a bit more complex in term of math. Like having a bad width/height ratio might also give something like that.