How to blur parts of a camera/canvas as an UI-background

Have you considered a post-process to do the filtering?

Alternatively, With a multi-camera setup, you’ll want to specify layerMasks for cameras and meshes. I will typically have a scene camera and a gui cam on different masks, with the gui cam in orthographic mode looking down on a similarly masked plane mesh with an AdvancedDynamicTexture that hosts the GUI controls. Full control over alpha and blending would provide you with an easy blur effect.

For your purposes, you could simply use CSS static or absolute positioning to position your UI elements in the appropriate places (cool UI design BTW!)

HTH

2 Likes

You may render the main canvas as screenshots (blobs), blur this image and pass for HTML element background. (Or browser will do with CSS basically the same with its own way).
In order not to kill performance I would make just in CSS something like opacity = 0.8 and very subtle noise overlay to imitate blur effect.

2 Likes

Hey @jelster
thanks for your reply. Yes, i’m using a post-process to achieve the blur. my fist approch was

this.camera_overlay = this.camera.clone();

var left = .7;
var bottom = 0;
var width = .3;
var height = 1;

var viewport = new BABYLON.Viewport(left, bottom, width, height);

this.camera_overlay.viewport = viewport;

var kernel = 128.0;
var postProcess0 = new BABYLON.BlurPostProcess("Horizontal blur", new BABYLON.Vector2(2.0, 0), kernel, 1.0, this.camera_overlay);
var postProcess1 = new BABYLON.BlurPostProcess("Vertical blur", new BABYLON.Vector2(0, 2.0), kernel, 1.0, this.camera_overlay);

this.scene.activeCameras.push(this.camera_overlay);

however this sadly dosn’t give me the desired effect… :frowning: The viewport only “narrows” the shown picture, while still centering the viewpoint of the camera. I want the exact same(!) picture as my original cam, only crop stuff away from it, that i dont want blured.

so i think i’m searching for a possibility to either crop/mask my cloned camera or my post-process… or something completly different… :slight_smile:

something like this :slight_smile:

oh and the fist screenshot isn’t mine. i think it’s from some jurassic park game, but i also liked the ui a lot :slight_smile:

1 Like

@labris i also thought about it. but it seams rather expensive to me. specially when i’m aming at 60fps and want to grad the user the ability to move while the ui is visible… there must be a way to achive this effect in “native” babylon?!?

1 Like

Thanks for the clarification. Would it be ok to only blur meshes in the affected area? Instead of multiCam and such you could make a material (using NME or shaders) to accomplish the blurring, perhaps

The Babylon GUI system is worth looking at for decent perf in ‘native Babylon’ - ignite help get where you’re trying to go too

2 Likes

sry, i’m afraid i’m not quite sure, what you mean. Put an extra material on the meshes I want to blur? Then probably not. What if a mesh is only partly in the affected area? Then this approach wouldn’t
work, or would it? It would either blur the whole mesh or not, but not half a mesh…?

I thought about using a box or plane with the size of the affected area, parent it to the camera and give it some „lense“-material, but i’m not sure if I could realize a nice gaussian blur effect this way like with BABYLON.BlurPostProcess…

In CSS I would just put the second camera or the post-process-„layer“(?) into a box, position and size it and then set the box to overflow: hidden; This is what I’m searching for, basically :slight_smile:

When logging the object generated by new BABYLON.BlurPostProcess I saw _parentContainer, but I couldn’t find anything on it in the API-Documentation…

Of course I’m open to using the BabylonGUI System, if it get’s me my box with overflow:hidden; :slight_smile:
(I just think, since the UI-„Content“ in my App is rather complex (Texts,Images,Videos, etc.), I probably better display the Content with HTML/CSS)
When looking at the AdvancedDynamicTexture documentation I saw, that one can define a background - but only with colors? Is there a possibility to use a camera-output as the background – and can I set overflow:hidden; ? :slight_smile:

In every path-based-graphics programm there is the possibility to subtract a form from another… Is there maybe something similar in Babylon? Something where I can subtract a form from the camera-output?

Sry, if i ask stupid questions, i’m fairly new to the 3d-world and this amazing framework. :slight_smile:

1 Like

I have done this by passing an ADT to a PostProccess along with the scene RTT. I did a big no no and made two ADTs one that was a mask and one that was the UI, but you can get away with one I found out later.

Once you have all your data in the Post you blur you main scene and then use the mask ADT to mix in the blurred data to the unblurred.

You can see the effect in action here:
https://mvrkracer.mvrk.co/

6 Likes

Hey @Pryme8

Thanks for your reply. Before i asked my first question regarding the blur-ui topic i also came across RenderTargetTexture but it seamed way to complex for my trivial(?) problem…

However pirating some code from https://www.babylonjs-playground.com/#KG6SGU#13 i was able to achieve the effect i want, using the GUI as @jelster suggested.

But… it seams soooo wasteful. And my FPS went down 20%…

Am i understanding correctly what i’m doing? :slight_smile:

By using renderTarget = new BABYLON.RenderTargetTexture(); and pushing every mesh in my scene into it I basically render my scene twice?

Then i use new BABYLON.BlurPostProcess() on my renderTarget to blur the whole, redundant scene.

Next i set um a BABYLON.GUI.Rectangle with a width of 0.333 and put my renderTarget as an image into it, basically throwing away two thirds of the just computed image.

There has to be a better, more performant way to achieve this! Somewhere i red, that the postprocess is like a pixel-layer? Then it must be possible to mask or crop it somehow…?

I already have the image generated by my camera. And it should be more performant to blur only part of the image rather then “clone” my complete scene, blur it an then crop it?

How would someone achieve something like this for example?

If there is no way to crop/mask a postprocess then there should be :slight_smile:

1 Like

Not gonna lie, I have no clue what’s going on in the PG you posted… but the process you described is about right, and you could always make a variance of what ever post process you want and include a mask/crop addition.

1 Like

Basically you need a full res none blurred texture (your rtt)

Two half res texture for a nice bidirectional blur (or only one texture for a kernel based one).

Then on the canvas you would first render the shape of the ui with planes at the position of your UI skinned with the blur at a position close to the camera (use an orthographic to simplify matching your ui position) then you render a fullscreen plane with the none blurred background on the back to prevent overdraw of the blurred area. This is about the most efficient you can go.

5 Likes

Here is a quick draft - https://playground.babylonjs.com/#WTJ7L1#1 (click on button twice to see effect).
Basically, the second camera should have the same properties as the main camera and render screenshots with blur postprocess for the whole canvas/viewport. The screenshot image with blur (or portion of it) will be the background of UI element.

3 Likes

This is the method I used exactly for the MVRK racer.

Thanks for your replies! I did a lot of testing and experimenting, but sadly I’m not quite there jet…

@labris. Thanks for the playground! I also tested your approach, but unfortunately when I want the blurry parts to update in realtime it seems to be not performant enough. :frowning: (same when I tryed DumpFramebuffer instead of Screenshot)

@Pryme8 & @sebavan:

I’m not quite sure about the second-camera-approch you guys described. How could I attach the RTT or the camera to the GUI this way? Or would I not do this and use the RTT as texture on a plane and set it as a “background” for the second camera, putting the GUI on top?

When setting up a plane and giving it a rtt-material it isn’t more performant, than the approach I used based on this playground https://www.babylonjs-playground.com/#KG6SGU#13 by @Gijs, where I am able to paint the RTT directly into the GUI. (Via: How to show a RenderTargetTexture on GUI?)

However, sadly, when I use this approach, I get this nasty edge-glitches… I assume it has something to do with the custom shader used…?

Even more sadly it appears I’m not competent enough to remove the whole greenscreen-custom-shader-thingie and just display the RTT without any shader :slight_smile:

This playground (https://www.babylonjs-playground.com/#5E96NN#2 // https://croncle.blob.core.windows.net/playground/renderTargetTextureImage.js) mentioned earlier in said discussion seams to be exactly what I want - unfortunately it is somehow broken, my firefox tells me “Uncaught DOMException: An attempt was made to use an object that is not, or is no longer, usable”… And again I’m not competent enough to fix it :slight_smile:

So my conclusion so far: it is possible to display a RTT within a GUI.
It also should be possible to set the image.source to a base64 output of the RTT (similar to what @labris suggested), but probably it is not very performant as a base64 would be rendert for every frame, correct?

As far as I understand @Gijs approach, the RTT doesn’t get rendert, rather transformed into a 2d-representation an integrated into the GUI?

1 Like

Because the image is blurred you may A) use lower resolution for RTT; B) render it, for example, every 4th frame to save performance with no harm for user experience.

This PG - https://www.babylonjs-playground.com/#5E96NN#2 - works in Chrome with Babylon 4.2 version.

2 Likes

Works with firefox as well when switching to BJS 4.2… I encounter the same problem on other playgrounds too… → https://www.babylonjs-playground.com/#10KKS5#5 (BJS4.2,ok // BJS5.0,broken)

can it be a BJS version problem with new ImageData() ?

1 Like

It seems that texture.readPixels returns a promise now, so using await or then fixes the error when using the current BJS version.
https://www.babylonjs-playground.com/#10KKS5#12

For the other playground thou, that does what you want, the external script would need to be updated to wait for the promise to resolve as well I think. Here I copied the script onto the playground and used .then() to wait for readPixels to finish.
https://www.babylonjs-playground.com/#5E96NN#6

This fixes the error, but IDK how good of a solution it is. Probably it would be better to call readPixels and putImageData less often than once per frame I think.

2 Likes

This is a change to support WebGL2 and WebGPU

3 Likes

yup it would be a background for the GUI but a foreground of the scene. not passing it to the gui is the best option to prevent expensive GPU → CPU → GPU resources sharing operations.

Yes it is normal cause in this case the GUI is rendered through WebGL not through the Canvas2D. The approach is actually fully similar to the other one.

Displaying the RTT without shader would mean introducing back to huge perf hit of sharing the resources across different contexts.

About the aliasing it is probably cause you are using a post process and you should turn on MSAA on your RTT by doin rtt.samples = 4;

3 Likes

Thanks a lot for all the replies!

So, again I did some experimenting…

@labris: I tried the render only nth frame with screenshot, but then the blurry area is (of course) async, which doesn’t look nice. This gave me a nice idea though: Whenever I don’t show the RTT in the GUI, I set the renderTarget.refreshRate to 1000, to save FPS and it works great. Thanks. :slight_smile:

@blake: I tested it and the performance of the canvas-approach in the fixed playground is not very good, as you suggested… (and I think I slowly begin to understand why this is and the differences between the two approaches…:slight_smile: Thanks a lot for fixing the playground anyways! (Once my elevator and the controls are fully ready I’ll invite you for a ride :slight_smile:

So I looked a bit more into the other (shader-based) approach and stripped it of all the parts I don’t need, removing the greenscreen-part and converting it into a position-based approach. Which works rather well, I’d say… I’m even able to raise the brightness of the affected area… Webgl is fun! :slight_smile:

Here is a PG —> https://playground.babylonjs.com/#P8HK81#10

However with every bit I understand, there are a lot more things I don’t unterstand :slight_smile:

This part should make sure, that the RTT (and the shader?) is only displayed within the borders of the „image“.
if(insideBox(orgPixel.xy) < insideBox(vUV)){}

But somehow the shader affects also the surrounding scene in a way that (as far as i understand) i didn’t put in anywhere. It makes the edges pixelated (which should be fixable via MSAA as @sebavan suggested) but also messes somehow with the emissive parts(?) of images/meshes. Here is a pic of how it affects my scene.

One can also observe the behavior in the playground (on top of the sphere). It goes away when switching off the guiTexturePostProcess in the inspector.
Can someone please explain to me why this is happening and maybe how to fix it?

Furthermore I lose the opacity of the GUI elements on top of the shader affected area and the button-text gets a pixelated black outline. How is this possible when they are on top of the shader area, or am I thinking CSS here? :slight_smile:

When I also plug in my DefaultRenderingPipeline I get even more strange results. Again in combination with the postProcess. (the Pipeline worked fine before)

The DefaultRenderingPipeline affects the whole Scene, the postProcess only camera or RTT? Jet I can recreate all the DefaultRenderingPipeline effects with postProcesses? What is the difference between the two? Why use DefaultRenderingPipeline?

Sorry for the (again) long post, I would offer a potato, but I don’t know if it’s customary around here… :smiley:

I hope at least my questions are getting slightly less stupid :slight_smile:

/////

I really enjoy this forum. It reminds me of the good old (intenet) days - before the invention of social media - when people just met online and discussed topics. Without screaming at each other… :slight_smile: Thanks for that!

5 Likes

I’m personally really attached to that as well :slight_smile:

2 Likes