Oculus Mixed Reality Capture with BabylonJS

has everyone ever tried out whether Oculus Mixed Reality Capture (MRC) works together with a babylonjs application?

MRC basically allows a video capture of a person wearing a VR headset in front of a greenscreen to be composed with a virtual image captured from the 3D scene from the virtual position that matches the camera in the real world. This gives the impression of a video capture from within the VR application but with the real video capture of the person.

I read from the docs of MRC that an application needs to support this. From a technical point of view, I get that basically the application needs a second camera to render the application from the point of view of the real camera capturing the person. This camera is send as a video stream to another computer that also gets the video stream from the camera with the greenscreen and composes one common video stream.

Any thoughts, any ideas about the topic?

Here is the original docs from Meta:

I have an Oculus Quest available but so far, did not bother to set MRC up in my appartment, but willing to do so by next week.

Best regards,

Hi @axeljaeger ,

interesting topic :slight_smile:

Babylon does support spectator mode, which is basically a different view rendering together with the two views that are the eyes. I can only assume the same way can be used for that.
Spectator mode is a pure helper-function. You can reproduce it differently, using a different reference space (or - you can change the viewer reference space to fit your needs).

This is the code for the spectator mode - Babylon.js/webXRExperienceHelper.ts at master · BabylonJS/Babylon.js (github.com)

Hello @axeljaeger just checking in if you would like any more help :slight_smile:

I also asked in the Oculus Developer Forum: Mixed Reality Capture with Browser based applicati... - Oculus Community - 969277

From my current understanding: An application has to explicitely support MRC. Oculus offers for both unity and Unreal a way to access some objects, I guess the position of the observer camera and then the developer basically has to render the scene from that position.

These objects need to be available in the oculus browser. I am not aware of any way to access them, so in the worst case, someone needs to build a minimal unitiy / unreal application, sniff the network traffic, figure out the communication protocol and then implement everything for babylon. AFAIK, the sourcecode of the MRC plugin for OBS is available as well:

so this might also be usefull.

I dont know when I find time to look into this.

Best regards,