How to share VR camera and controller position and rotation with other players?

I’m new to babylonjs and 3d in general so forgive the naive question. I’m attempting to create a multiplayer WebXR game where I would need each player to be able to “see” each other’s 3d avatar.

I’ve setup a local server with websockets so I am capable of broadcasting and receiving json messages from each player. (This is just a proof of concept, I know I’ll probably need to use something more efficient like webrtc with UDP later)

For now, I’d simply like to be able to render a box mesh representing the other player’s head and maybe boxes for each hand controller they are holding.

  1. How do I get each player’s headset absolute position and orientation? From my research so far, I think I can access the headset camera like this:

       this._scene.onBeforeCameraRenderObservable.add((cam) => {

    Once I have the camera object, there is something called cam.globalPosition which sounds promising, but there is no globalRotation. There is also something called getWorldMatrix, which seems promising, does that give me both position, rotation and scaling in one object?

  2. The second part of the question is, once I can get that data from player1, and I draw simple mesh box to represent their face, how do I apply the data (e.g. worldmatrix?) to the mesh so that the mesh is continually updated (translated, rotated) when the other player moves their head?


This is more a general babylon question rather then a WebXR question, as this applies to any babylon-based game/app.

  1. The camera’s position is almost (if not all the time) in world/global coordinates. Unless you parent the camera (which i wouldn’t recommend in XR) the camera’s position and its rotation(quaternion) are global. there is no real need to use the onBeforeCameraRender observable, you can do that in any interval you like. The scene’s active camera is stored at scene.activeCamera. When in XR, it is the activeCamera.
    The world matrix is a matrix holding the entire transformation. You will need to know how to decompose it on the other side, but yet - is should work correctly.

  2. And we come to decompose! :slight_smile: If you send position and rotation, it is simple - just apply it to the mesh. If you use a matrix, you can use its decompose methos to get scaling, rotation(quaternion) and translation, which you can apply to the mesh on the other side