Intercept VideoDome frames

Questions
1

Hi friends,

Above is my Playground code to intercept frame data of a VideoDome and try to turn it from colored to grayscale video. My ultimate goal is to perform some on-the-fly video effects frame by frame.

Somehow my code does not work. The problems I am having are:

(1) Shader code never got executed. The video remains colored
(2) WebGPU seems to crash after 2-3 minutes

Can you help me to overcome the problems please ? Thanks

2

You may achieve what you want with the help of 2D Controls - Babylon.js docs

3

Thanks @labris. I have had a look at 2D controls which indeed offers the grayscale effect. However, my ultimate goal is to perform some more intensive/complicated video processing, that’s why I need to leverage WebGPU which 2D Controls do not support.

My use case requires frame interception directly on Babylon.js with WebGPU and compute shaders to have more flexibility and computational power, rather than relying on 2D control wrappers and the old WebGL.

4

Do you need to apply your effect on the source full texture or on the output only ?

doing it on the output only has a nice side-effect of only processing what you see but might also lead to more processing depending on your texel to pixel ratio.

5

Hi @sebavan , I want to apply on the source texture.

6

Thanks, let s check with @Evgeni_Popov when he ll be back next week :slight_smile:

In webgl I would rely on an effect wrapper with an effect renderer for similar cases not involving compute.

8

There are a few problems in your PG:

  • you create a new texture each frame, that’s why it crashes after a few minutes with an “out of memory” error
  • you should not pass an InternalTexture to ComputeShader.setTexture, but a BaseTexture instead
  • dispatchWhenReady will execute after the end of the frame, in a JS micro-task. That’s better to have it executed inside the frame, so use dispatch instead. To make sure you call it after the compute shader is ready, you can check ComputeShader.isReady() in a setTimeout (see the fixed PG below)
  • mesh.onAfterRenderObservable is a better observable than engine.onBeginFrame

Also, you can’t update the diffuse texture of the video dome with your new texture, as the diffuse texture is the video texture. What you can do is to create another video dome and apply your texture to this dome. As you are not interested in displaying the first dome, you can set its position out of the screen, but you will need to set mesh.alwaysSelectAsActiveMesh = true to make sure the video is still processed.

PG with these fixes:

It may be interesting to get rid of the first video dome, as you only need it to advance the video texture, and generate the texture yourself. See how it is done in this PG:

1 Like