VideoTextures from device camera crash on iOS but I figured out how to fix it

Greetings ya’ll,

I had been struggling with this for a long time, and now that I fixed it I want to get it into the BabylonJS codebase, but the Github said to post here first, so here I go…

VideoTexture.CreateFromWebCam creates a video texture from the device’s camera. However, I found that the video stream would either freeze, or start up completely black on iOS very frequently. It was a huge show-stopper. This includes all iPhones and iPads (technically iPadOS now, but whatever.)

If you don’t know how camera video streams work, they basically have to be assigned to a <video> element.

VideoTexture.CreateFromWebCam eventually calls VideoTexture.CreateFromStreamAsync, and this function actually creates the <video> element to host the video stream.

The solution to the iOS problem is this:

I discovered that appending the <video> element to the DOM fixes the problem. VideoTexture.CreateFromStreamAsync creates the element but never actually appends it to the DOM, for obvious reasons. I think we should change this so that this video functionality isn’t fundamentally broken on iPhone/iPad.

If you check out this thread which gave me the clue for this fix, another person says that when you add the <video> to the DOM, you can’t hide it with display: none; or else the bug will come back. But in my app, I simply have another element covering up the <video> so it’s no big deal.

I don’t know the correct way to add this to BabylonJS because I’m not sure how opinionated BJS should be about where exactly this video element gets popped into the DOM. Additionally, this video element must be removed when the BJS engine is disposed or the scene is disposed.

Anyways, there is nearly nothing out there on the internet about this bug that I could find so I’m hoping this post helps someone else.

By the way, because this fix is not actually in BJS yet, I fixed it in my app by actually copying the following functions out of node_modules/@babylonjs/core/Materials/Textures/videoTexture.js and implementing them as standalone functions in my codebase;

  • CreateFromWebCam
  • CreateFromWebCamAsync
  • CreateFromStreamAsync

I tweaked CreateFromStreamAsync to add the <video> element to the DOM where I wanted it. I also had to remove references to _this and Logger which weren’t necessary anyways. So it’s a hack but the functions live pretty well outside of the VideoTexture class and overall it was pretty easy.

So, please let me know how we can fix this for the rest of the world.

1 Like

I agree on that. But maybe we could add an optional parameter to ask the system to add the video element to the DOM. Or we can also return it

But maybe we could add an optional parameter to ask the system to add the video element to the DOM. Or we can also return it

I’ve given this some more thought. Because this issue is an iOS compatibility issue, I suggest that the video element is added to the DOM automatically. Where it’s added doesn’t really matter if we provide the necessary CSS styles to minimize its impact on the UI. Otherwise, if BJS simply returns the video element, the developer would have to know about this issue and how to solve it in order to avoid it, and also handle the video element differently depending on the current device’s platform.

I suggest that we check for iOS (not sure the best way to do that) and then automatically append the video to the DOM on iOS.

The result of this is that iOS users will see a video pop into their app.

We could also automatically append a short <style> tag that hides the video element without using display: none; Maybe we could even include those styles inline with the video element itself.

I just got a reply an hour ago with some styles that can be used to ‘hide’ the video: Canvas video stream freezes on mobile safari · Issue #14 · jeeliz/jeelizFaceFilter · GitHub

that could work…fancy doing a PR?

Yeah sure! I’ve never contributed to something like BJS before and I’m a bit intimidated to do so, but I’m willing to try my best! What’s the correct way to get started? :cowboy_hat_face:

No need to be intimidated. We all started from scratch at some point :slight_smile:
Here is a doc to get you started: Start Contributing to Babylon.js - Babylon.js Documentation

1 Like