How to share VR camera and controller position and rotation with other players?

Questions
1

I’m new to babylonjs and 3d in general so forgive the naive question. I’m attempting to create a multiplayer WebXR game where I would need each player to be able to “see” each other’s 3d avatar.

I’ve setup a local server with websockets so I am capable of broadcasting and receiving json messages from each player. (This is just a proof of concept, I know I’ll probably need to use something more efficient like webrtc with UDP later)

For now, I’d simply like to be able to render a box mesh representing the other player’s head and maybe boxes for each hand controller they are holding.

  1. How do I get each player’s headset absolute position and orientation? From my research so far, I think I can access the headset camera like this:

       this._scene.onBeforeCameraRenderObservable.add((cam) => {
     ...
   })

    Once I have the camera object, there is something called cam.globalPosition which sounds promising, but there is no globalRotation. There is also something called getWorldMatrix, which seems promising, does that give me both position, rotation and scaling in one object?

  2. The second part of the question is, once I can get that data from player1, and I draw simple mesh box to represent their face, how do I apply the data (e.g. worldmatrix?) to the mesh so that the mesh is continually updated (translated, rotated) when the other player moves their head?

Thanks!

2

This is more a general babylon question rather then a WebXR question, as this applies to any babylon-based game/app.

  1. The camera’s position is almost (if not all the time) in world/global coordinates. Unless you parent the camera (which i wouldn’t recommend in XR) the camera’s position and its rotation(quaternion) are global. there is no real need to use the onBeforeCameraRender observable, you can do that in any interval you like. The scene’s active camera is stored at scene.activeCamera. When in XR, it is the activeCamera.
    The world matrix is a matrix holding the entire transformation. You will need to know how to decompose it on the other side, but yet - is should work correctly.

  2. And we come to decompose! :slight_smile: If you send position and rotation, it is simple - just apply it to the mesh. If you use a matrix, you can use its decompose methos to get scaling, rotation(quaternion) and translation, which you can apply to the mesh on the other side

2 Likes
3

Thanks for the tip about scene.activeCamera. I suppose I can get the data for an avatar’s head movement using:

scene.activeCamera.absoluteRotation
scene.activeCamera.globalPosition

But how do I send this data to a server on an interval or when there are changes? From my research so far there is:

scene.registerBeforeRender

which is called about 60 frames before the render. But I’ll have to write my own logic to find out what changed. Is that callback the recommended one to lookup and publish changes? Or is there someplace better to get and send camera position/rotation changes?

Also, I haven’t found where I can get the two controllers (left and right hand) position and rotation information yet. Can you please tell me where in the scene I can locate those data? And also where is the best callback or hook for me to gather and send that data? Thanks!

4

You can find the controller callbacks/position/everything else you need right here - WebXR Input and Controller support - Babylon.js Documentation

About sending the data to the server - this depends on the server you are running and the interval you want to update. Updating 60 times per second means that the request should take less than 16ms, otherwise there will be a delay with the next update to the server (doesn’t have to be, depends on the implementation). But this is really something you need to check yourself and see what fits. Or maybe use some socket-server that will run the updates instead of you dealing with the synchronization.

5

Hey @owen, did you find your answer? I’m trying to achieve something similar to you. I’m finding that the camera.globalPosition and camera.rotationQuaternion don’t quite match up with the coordinates of the scene. Here’s a playground to expain a bit more… https://playground.babylonjs.com/#L3MG2Y#4

6

yes I did. I ended up setting up an onViewMatrixChangedObservable on the camera and then sending cam.position and cam.absoluteRotation as arrays over the wire using websockets.

I’m not sure if what you’re doing in your playground makes sense to me. For some reason setting the elf mesh position doesn’t do anything, but if you move it’s parent, the “root” then it will be moved to the camera position.

7

Oh yes, using the parent works! I still have quite a bit to learn about the relationships of nodes and parent nodes in Babylon