I’ve encountered a significant FPS drop issue while utilizing a MediaStream in a video element with a VideoDome, particularly on macOS and iOS Safari. The MediaStream is configured to contain both audio and video tracks, with the video track added approximately 1 second after the audio track.
The sequence of steps is as follows:
Create a VideoDome with a video element.
Add an audio track to the MediaStream and assign the MediaStream to video.srcObject.
Add the video track to the MediaStream and assign the MediaStream to video.srcObject.
However, an error arises when adding the audio track without the video track, accompanied by the following error messages:
WebGL: INVALID_VALUE: texImage2D: no video
WebGL: INVALID_VALUE: texImage2D: no canvas
Upon inspecting the source code here, it’s evident that this._videoTextureSupported becomes false. Additionally, an extra canvas is created and drawn in every frame.
This issue seems to be particularly pronounced on macOS and iOS Safari, leading to a significant drop in FPS.
I’d greatly appreciate any insights or suggestions on resolving this, especially if anyone has encountered a similar challenge on these platforms.
Hi and welcome to the Community,
Looks like you uncovered something here. I believe cc @RaananW would be the person to contact since (as far as I remember) he’s been looking into video components.
@RaananW
I’ve attempted to reproduce the issue in the Babylon Playground. You can find the playground example here: Reproduction Playground.
In the actual scenario, I receive video and audio tracks from WebRTC. However, for the sake of this playground example, I’ve created a video track with a canvas in the same resolution, and the audio track is obtained from navigator.mediaDevices.getUserMedia.
In the playground example, there are two functions: setupTracks_audioFirst and setupTracks_videoFirst. The issue with FPS drop occurs when using setupTracks_audioFirst.
I would greatly appreciate your insights and guidance on whether this issue is specific to certain browsers or if there are specific conditions that trigger the FPS drop.
There seems to be an issue with iOS and captureStream. You set the FPS of captureStream to 5, which have caused an interesting issue with the update process of the video texture. remove capture stream’s FPS definition (or setting it to 0) will do wonders on mac OS’s safari. iOS, however, is a different beast altogether.
As this is not directly a video html element, we can’t provide the video element to the gl context. We need to create a canvas, draw the image captured on that canvas, and darw using this canvas. This clear + draw might be faulty on iOS, because when removing the first parameter on iOS it doesn’t display the video. Might be related to this - javascript - How to detect when `canvas.captureStream()` is not supported? - Stack Overflow , which seems to be a long standing iOS issue.
Thank you for your response and the helpful information. I’ve observed that you commented out requestAnimationFrame(run). However, even after removing or setting the FPS definition of the capture stream to 0, the issue of low FPS still persists. It appears that the suggested fix didn’t have the desired effect. Playground
Regarding the iOS-specific issue you mentioned, it seems that canvas.captureStream() is now functionally operational on iOS. This suggests that feeding the MediaStream into a video element should work fine. Could you please confirm if this understanding is correct or provide further insights into this matter?
I also attempted to replicate the issue by reproducing the canvas clear + draw process in every frame. Interestingly, I didn’t encounter the low FPS issue on iOS during my tests. It seems that this specific problem might not be consistently manifesting, or perhaps there are other factors at play that I haven’t accounted for yet.
Could you provide any additional details or suggestions that might help in accurately reproducing this issue?
I don’t have a mac, so I can’t really debug this. I have asked some team members who do use mac to check those playgrounds and have reported what their experience was.
I did check it on windows (firefox/chrome) and on android (chrome/brave), and it did work correctly. Removing the FPS definition made it work on a mac (reported ), so it does seem to be an iOS issue. it might be a race condition between update and read, it might be an issue with decoding videos on the os itself, it’s just really hard to debug if i can’t debug the actual browser that has issues.
It all comes down to the texture update in the engine’s video texture. if it was possible for me to debug, I would check the amount of time needed to run the update video texture function on iOS - I assume this is the function that causes the delay. Then I would dive into this function and try to understand what part of this function takes time and where are we blocked.
After some research, I found a WebKit bug report (234920 – ImageBitmap created from a video element has poor performance) that might be relevant to your issue.
It discusses how createImageBitmap could cause rendering problems in Safari, particularly when transferring data between canvas contexts. This seems to align with scenarios where a video element lacks a video track, leading Babylon to create an extra canvas for texture. Additionally, there’s a demo (Video imageBitmap: Webkit Bug 234920) illustrating the low fps issue.
It would be helpful to compare this with my problem to confirm if it’s related.
), so it does seem to be an iOS issue. it might be a race condition between update and read, it might be an issue with decoding videos on the os itself, it’s just really hard to debug if i can’t debug the actual browser that has issues.