3D XR Recording Camera

I’d like a 3D XR Recording Camera. It would record (render) the 3D scene in action as a 3D stereoscopic equirectangular mp4 file. The location of the camera and the direction it faces would be programmable as the application runs. It could be a 1st person camera, or a 3rd person.

Type - 1) It would be nice if it were real time, but it could just record a sequence of still frames which could then be combined to a video.

Type - 2) It would be able to record in real time and be exported to a video editing console or compositor to compose video from multiple sources in real time.

Type - 3) It would be able to run in a webXR scene created by some other application.

It each case, it would be the same as a camera that sends stereoscopic data to a headset, except that it would record the entire 360 x 180 (4Pi) Dome and not just the FOV for the HMD.

Some use cases would be:

a) Use Babylon to create ​3D 360 VR videos.
b) Use Babylon to create 3D 360 videos or live streams which incorporate volumetric video from a separate stream.
c) Record events in webXR. For example, an event in a webXR social application like Hubs, or in a social application native to Babylon.js.
d) Record the experience as a person plays a game (or visits an experience) written in Babylon.

This kind of functionality seems to available in Unreal, probably in Unity, and definitely (at least Type 1 – see link) in Blender.

How to Render a 3D VR Video from Blender
Blender Tutorial: How to Render 3D VR Video from Blender | VeeR VR Blog

It could be very powerful and useful in Babylon.

1 Like

@RaananW knows all about our XR stack so he might have some ideas.

This is less about XR and more about generating a video from a 3D scene that can be viewed in the video dome (for example), and of course in XR too.

It’s a nice feature to have, I totally agree. This is something we can think about for future releases, would you mind adding a feature request on github referencing this thread when commenting?

1 Like

Hello, and just looking at this now.

Is there a way to take the composited XR frame and render it to video? The VideoRecorder and VideoRecorderOptions do not have support for pointing cameras or frame buffers at them.

How is this otherwise being done?

r

Blender and Unity (and probably Unreal) can do this, I think. Unity last year introduced several interesting features for cinematic XR creators.

Several steps are required.

  1. A 3D stereo camera should be controlled by software with options to set its recording properties: (1) position and direction, (2) cubemap; Or equirectangular (360, 180, or other FOV (vertical and horizontal)) ; Or framed; (3) top/bottom Or left/right; (4) still Or video (5) Usual camera options: frame rate, depth of field, focal point, and f stop; other?

  2. The camera should be able to output and save in various video and still formats, such as mp4, pro-res, jpg, etc

1 Like

I think Quest has this as an option now. The BIG extra is audio, including microphone, is included.

2 Likes

As far as I know, Quest can record stills and audio/video, but only what you see in your headset, and only 2D. 3D VR video capture from a programmable 3rd party camera is a feature I and others have been asking for for a long while. (If it does exist, please let me know how to do it!!!) There is even a FB group now for “Metaverse Filmmakers”, but the films made in Quest, so far, are all 2D, with an Avatar held camera.

I do not see it either when I go to start a new recording, nor in experimental features that can be turned on, but I saw a tweet in the last month with a screen shot. On it was a list of options, that showed combinations 3D / 2D, along with frame rates & I think aspect ratio combinations supported.

I did find this, which describes what I am SURE I was, but cannot find.
Meta Finally Adds Quest Video Record Settings For Developers (uploadvr.com)

Follow up:
I got a re-tweet of this last month

I have a developer ID at Oculus. I installed Oculus Developer Hub. It is clearly for Unity Devs, but there are things which are useful. I still have not gotten any video setting working.

1 Like

Very interesting! Thx!