Oculus Mixed Reality Capture with BabylonJS

has everyone ever tried out whether Oculus Mixed Reality Capture (MRC) works together with a babylonjs application?

MRC basically allows a video capture of a person wearing a VR headset in front of a greenscreen to be composed with a virtual image captured from the 3D scene from the virtual position that matches the camera in the real world. This gives the impression of a video capture from within the VR application but with the real video capture of the person.

I read from the docs of MRC that an application needs to support this. From a technical point of view, I get that basically the application needs a second camera to render the application from the point of view of the real camera capturing the person. This camera is send as a video stream to another computer that also gets the video stream from the camera with the greenscreen and composes one common video stream.

Any thoughts, any ideas about the topic?

Here is the original docs from Meta:

I have an Oculus Quest available but so far, did not bother to set MRC up in my appartment, but willing to do so by next week.

Best regards,

Hi @axeljaeger ,

interesting topic :slight_smile:

Babylon does support spectator mode, which is basically a different view rendering together with the two views that are the eyes. I can only assume the same way can be used for that.
Spectator mode is a pure helper-function. You can reproduce it differently, using a different reference space (or - you can change the viewer reference space to fit your needs).

This is the code for the spectator mode - Babylon.js/webXRExperienceHelper.ts at master · BabylonJS/Babylon.js (github.com)