VideoDome clickToPlay and dispose maybe not as expected

Bugs
1

This is minor bug/issue/clarification, and it is not specific to 5.x.

I ran in to something unexpected using VideoDome, and while researching that, I also saw a potential minor enhancement. I can post these as bugs/features in Github if useful, but I wanted to post here for discussion first in case I am missing something.

1) Bug/Unexpected - VideoDome.clickToPlay
Here is a PG showing what confused me:

If you press ‘d’ key the video dome texture will disappear, however, if you click the blank canvas again you hear the audio from the video start playing again. The reason is that clickToPlay=true causes VideoDome to register a global scene.onPointerUp callback (source), and it is never unregistered. The underlying video element is never fully removed from the DOM (more on that below), and so the global callback starts playback again even though there is no visual video output surface.

Here is PG showing that deleting the global callback prevents the issue:

The correct fix would be to have VideoDome.dispose unregister this callback.

  • A fix should probably also switch to scene.onPointerObservable instead of global callback.
  • Less important, I would suggest that the onPointerUp listener should be an addOnce listener. As it is, the video will play/restart on any and every scene pointerUp event. This might be controversial though, and might have non obvious implications for devices that require user event context before playing. (Some previous discussion.)

I believe this bug to not be important beyond the confusion it might cause, because it can be easily avoided by setting clickToPlay=false and triggering video play manually.

2) Potential Enhancement - VideoDome.dispose and Video Element
It might be assumed that calling VideoDome.dispose would also release the underlying Video element and associated media resources in the browser, but that is currently not the case. That is evident given the previous issue where even though VideoDome.dispose is called, the video element can still be played.

Looking in to this, it seems that getting browsers to release Video resources is not straightforward and browsers aren’t always consistent about it. Based on some SO posts and similar, I came up with this PG to show how it might be done:

Code like this might make sense in VideoTexture.dispose. However, I think testing across various browsers/devices would be important.

(p.s. For anyone testing around this, using these PGs, I’ve found on Chrome/Windows, sometimes the audio stops playing after multiple PG plays or page loads. That can be confusing when you are trying to check if the underlying video playback has stopped. I suspect this is related to the dispose not fully disposing and so browser eventually running out of some low-level resource. Restarting browser usually resolves.)

2

I ll fix this ASAP cause it is also a source of potential leak…

Yes I ll do it as well in a couple hours.

3

Here it is Fix VideoDome Dispose by sebavan · Pull Request #12127 · BabylonJS/Babylon.js · GitHub, the texture is not disposed automatically as it needs to be specified as a parameter of the dispose function like the other meshes and I do not want to break back compat on this one :slight_smile:

4

Cool. Thanks!

And yeah, makes sense on disposeMaterialAndTextures. Using my original PG, calling dispose like this:

  • videoDome.dispose(false, true);

…prevents the issue. I don’t believe it’s actually letting go of the underlying video element (?), but that’s probably minor and common.

Additionally, indirectly related to this specific issue, it might be worthwhile to add to the VideoDome docs information on how to manually trigger play? It took me some time to figure it out:

        // When autoPlay=false...

        // This does not work. Starts to play then stops:
        // videoDome.videoTexture?.video.play()

        // Unexpectedly, neither did this. Sometimes black, sometimes not, but never plays:
        // videoDome.videoTexture.video.addEventListener('canplay', () => {
        //     videoDome?.videoTexture.video.play()
        // })    

        // This works and should probably be added to docs:
        videoDome.onLoadObservable.add(() => {
            videoDome.videoTexture?.video.play()
        })

Related PG:

5

Yes sure would you be willing to make a PR to the related doc page 360 Video Domes | Babylon.js Documentation ?

6

As noted in Kaliatech’s post, ClickToPlay = false works. But “ClickToPlay” is confusing, and true is the default. There should be a new boolean (for the moment I call that “ManualPlay”) that does almost the same thing except for swapping the true and false. “ManualPlay = true” should be the default. You can then control the starting and pausing with another button. The “new name” should be clear that “NewName= true” means that the video needs to be started and stopped by another button. It should also be clear in the documentation what happens when “NewName=false” is used. And how to dispose (or ortherwise replace or get rid of) the Dome and/or the video.

That is separate from the question of how best to play a succession of different videos with the same VideoDome, or other VideoDomes quickly, efficiently, and without leaking resources/memory (in any browser). As noted in the post, dispose has issues dealing with the video texture. I saw there was some work done on this but it is not clear in the discussion exactly what was done or how that works now. (I do not have access right now to anything but this email of the post.) How exactly does it all work now? Is the documentation changing?

7

I’ll submit a PR with some minor doc improvements and clarifications around this. However, I ran in to another issue related to autoPlay and autoUpdateTexture while trying to do so. I’ll post a new topic for that in a few minutes.

I’m not sure this could be done while still maintaining backwards compatibility, and that variable would seemingly be redundant with the existing autoPlay variable.

I agree that autoPlay is problematic. I suspect some of the confusion is due to browser policies changing over time. AFAICT, chrome’s policy now is to allow autoplay only IF the audio is muted. More info. And looking in to Babylon source, can see that VideoTexture will try to automatically mute the audio if needed when autoPlay=true. (Which surprised me.)

The previous change sebavan made did not change how dispose works. He addressed the issue of the onPointerDown listener leak. But as noted, if you call VideoDome.dispose(false, true), the second parameter (disposeMaterialAndTextures) will ensure the underlying texture resources are disposed and things generally work as expected. HOWEVER, that currently does not remove the underlying video element from the DOM and it doesn’t clear the video’s src. So I would assume that adding/disposing multiple VideoDomes (and/or VideoTextures) would eventually lead to out-of-memory or similar type problems. However, I have not verified that and it maybe becomes questionable as to if this is really in the domain of babylon.js. For those that have these type issues, the PG I posted shows how the video element can be completely disposed. (In theory…as browsers are not all consistent about this.)

8
9

Yup only muted video can autoplay from browser policy.

Yup I ll do that now.

10

“ManualPlay = true” should be the default.

I’m not sure this could be done while still maintaining backwards compatibility,

The reason I suggested a new “redundant” variable is so that ClickToPlay would remain, and anyone using ClickToPlay would find no change. Defining “ManualPlay” might automatically set ClickToPlay to the opposite of “ManualPlay”, or otherwise automatically override ClickToPlay. Simply renaming ClickToPlay to convey how it actually works might also help.

On another topic: The video element is given by

  • textureUrlOrElement: string | string | HTMLVideoElement

    defines the url(s) or the (video) HTML element to use.

<<<

I’m not quite sure how this works if there are URLs and not a single URL.

Curious: What are you working on with VideoDomes? What company are you with?

Thanks for your work, and your replies!

Eric

11

The HTML video element supports multiple sources, in which case, the browser is supposed to select the first one it thinks it can play. Multiple sources are provided as child elements, and you can see how BabylonJS does that here depending on what it is given:

Currently not affiliated with any company, and I should probably get a real job again soon. Currently, I’m just working on ideas I find interesting. I’ll probably submit some low-key demos in a week or so. My interest in VideoDome is for use in a WebXR environment. An improved implementation of VideoDome will rely on WebXR Layers. That said, AFAICT, there’s not a big market for VR video and interest has died down considerably. I do predict VR180 is going to make a small come back though. And, my predictions are only wrong 50% of the time. :slight_smile: It could also be that I’m looking for reasons to justify purchasing a Canon EOS R5 w/ the Dual Fish Lens.

12

Hi–

Thanks for the reply

the browser is supposed to select the first one it thinks it can play.

That’s what I thought it did, but I do not understand the use case. Why would you give a list of URL’s and let the program figure out which is the first one that works? Or… Is it constructing a playlist, that will automatically play one video (that woks) after another (that works)?

AFAICT, there’s not a big market for VR video

Actually, I think there is (or will soon be) a huge market. But there has been no good way to display VR. Neither FB nor YT work very well with VR… So far, no other VR player has great traction. Worse, currently there are no good pro-sumer cameras (now that Vuze is aging, out of stock and out of business).

That is about to change.

a) Kandao will release QooCam Ego at the end of the month. It is a ~$400+ 3D (FOV < 180) consumer camera and viewer.

b) Canon is coming out with the R5c in early April, which will (hopefully) fix the video problems with the original R5, making low-end Pro 3D180VR fun again. More expensive 360 3D VR and volumetric capture rigs do exist for Pro projects.

c) QPORIT is currently testing a comprehensive suite of (mostly Babylon.js based) tools to create webXR websites which will allow users to easily integrate all kinds of VR and non-VR media on their own webXR site.

Eric

13

It is not a playlist. The idea in the HTML spec was to make things easier on web developers. Consider that different devices support different video formats/codecs. So when delivering video for multiple devices, developer might include multiple sources like:

  • video.h265-hevc.m4v (video/mp4; codecs=hevc)
  • video.av1.webm (video/webm; codecs= avc1)
  • video.h264.mp4

The developer can rely on browser automatically picking the first (best) one in that list it can play which it will then download. Developer didn’t have to sniff the device. (In practice, it’s complicated because codecs usually have to be specified too, and I would guess most advanced users of this in Babylon would probably pass in the the video element instead of specifying sources as a parameter.)

Funny you mentioned Kandao and QPORIT as I have been testing with videos of both of theirs the last couple hours. (I didn’t realize it was QPORIT, but I recognize the guy now from the video.) I think there is a bug/issue with Babylon’s handling of 180 in SIDE-BY-SIDE mode, but it’s difficult to be sure because seems everyone encodes/formats these videos differently.

14

Kandao QooCam EGO, out of the camera is a 3D flat video. It is not equirectangular. It is not 180, the FOV is less. For YT, they have been adding metadata but It is not clear if they convert to equirectangular.) Where are you getting QPORIT videos from? They are probably equirectangular, created in VUZE (360) or VUZE XR (180). Some may be flat, created from a flat 3D camera.

What bug/issue are you finding with SBS? For Babylon VideoDome, what are you using to set half-mode? There may be a difference in whether you set it to half-mode within the Dome or after the dome is created. Also, Because EGO is flat and not 180, it can not be used the same way as equirectangular 180. You need to add a mesh.

15

Thank you. All good info and confirms much of what I’ve seen.

I believe I found one in an old PG. I believe it’s from QPORIT only because seems to be same person on the QPORIT facebook page.

I’m not yet positive it’s a bug. I only know that I have not been able to make a decent quality 180 stereo video dome yet. Part of the problem is finding high quality 180 stereo content. I have been trying to make use of the sample Canon released to youtube here:

I had to download it from YouTube though, so it’s not like I have the raw footage. From YouTube, it appears to be be 180 Side-by-Side, equirectangular. However, playing back in Babylon.js, the halfDome=true seems to only show half the video (though it is in stereo) and with possibly incorrect aspect ratio. Hard to tell though. I’ve tried multiple re-encoding options with ffmpeg at this point.

I’m probably going to give up for a while as this video is really hard to test with and I haven’t been able to find other high quality VR 180 stereo videos yet. Not sure what Canon was thinking. I’m hopeful Hugh Hou will release raw downloads of some of his recent tests. (He released to Oculus TV, but I haven’t found a way to get those raw files even when cached.)