Introducing: Smart Filters Preview

[amoebachant]

Hey folks, we’ve been hard at work on a brand-new Babylon feature that we’re calling Smart Filters. We plan to officially “release” Smart Filters with Babylon.js 8.0 at the end of the month, however we wanted to give our faithful and trusted community a sneak peek at it to get your early thoughts and feedback!

So, what is this exactly?

Babylon Smart Filters builds on our rich library of node-based graph tools, unlocking some entirely new capabilities. Smart Filters is a non-destructive, node-based system for building shader based 2D effects for scenarios like real-time video filters. We call them smart because this feature includes an optimizer which automatically minimizes the number of draw calls and intermediate textures for you.

You can build Smart Filters programmatically:

Or visually using a new tool, the Smart Filter Editor:

Blocks themselves can be defined in code, or you can import specially annotated GLSL files as new blocks you can use.

How to try it out

Preview warning: Remember, this is in preview form, so you shouldn’t take a dependency on it yet until we’re ready to take it out of preview later this month.

That said, we’d love your feedback on this sneak peek!

• A great place to start is the new Smart Filter Editor where you can play with some demo blocks to get an idea of what’s possible: https://sfe.babylonjs.com
• You could then dive into the code and check out the demo environment which shows how you could implement Smart Filters into your own project.
  • Clone the @babylonjs/smart-filters repo
  • npm install
  • npm start
  • and away you go!

Send us your feedback

Leave any thoughts or feedback you have on the preview here in this thread - it’ll be helpful as we get this feature ready for release. We look forward to hearing from you!

[cx20]

Thank you for providing this interesting feature.
I tried it out, but am I using it correctly?
Is my understanding correct that custom GLSL needs to be written in a somewhat unique format?

Smart Filter Editor

// { "smartFilterBlockType": "SerializedRippleBlock" }

uniform sampler2D input; // main
// { "default": 0.5 }
uniform float time;
// { "default": 10.0 }
uniform float frequency;
// { "default": 0.03 }
uniform float amplitude;
// { "default": 0.5 }
uniform float centerX;
// { "default": 0.5 }
uniform float centerY;

vec4 mainImage(vec2 vUV) { // main
    // Calculate the distance from the center point
    vec2 center = vec2(centerX, centerY);
    float dist = distance(vUV, center);
    
    // Compute the ripple effect
    float wave = sin(dist * frequency - time * 5.0);
    
    // Displace the original coordinates based on the ripple effect
    vec2 offset = normalize(vUV - center) * wave * amplitude;
    
    // Attenuate the effect as the distance increases
    float decay = max(0.0, 1.0 - dist * 2.0);
    offset *= decay;
    
    // Sample the texture from the displaced coordinates
    vec4 color = texture2D(input, vUV + offset);
    
    return color;
}

[Heaust-ops]

wait this is for post processes?
This has the potential to be basically Da Vinci Resolve on the web
at least for color grading!

this is so cool
absolutely revolutionary :3

if we also get the ability to add custom nodes and a place to upload it.
think of the possibilitiez, you don’t need to understand a line integral convolution you can just use a community node for it!
it’ll absolutely bring the raw power of post processing to hobbyists and just build a world of creativity

Considering everything, I think this might be my absolute favorite thing this release by far!
Thank you so much for this : DDD

[cx20]

I created a GPTs to prepare GLSL code with meta-information for Custom Blocks.
It seems to be functioning reasonably well.

GLSL to Smart Filter Converter

ChatGPT - GLSL to Smart Filter Converter

Smart Filter Editor

[kzhsw]

Since node materials can already be used with post processes, this can be integrated into node material, to enable usage as post processes or material

[amoebachant]

@cx20

// { "smartFilterBlockType": "SerializedRippleBlock" }

uniform sampler2D input; // main
// { "default": 0.5 }
uniform float time;
// { "default": 10.0 }
uniform float frequency;
// { "default": 0.03 }
uniform float amplitude;
// { "default": 0.5 }
uniform float centerX;
// { "default": 0.5 }
uniform float centerY;

vec4 mainImage(vec2 vUV) { // main
    // Calculate the distance from the center point
    vec2 center = vec2(centerX, centerY);
    float dist = distance(vUV, center);
    
    // Compute the ripple effect
    float wave = sin(dist * frequency - time * 5.0);
    
    // Displace the original coordinates based on the ripple effect
    vec2 offset = normalize(vUV - center) * wave * amplitude;
    
    // Attenuate the effect as the distance increases
    float decay = max(0.0, 1.0 - dist * 2.0);
    offset *= decay;
    
    // Sample the texture from the displaced coordinates
    vec4 color = texture2D(input, vUV + offset);
    
    return color;
}

Hi @cx20, cool block! Yes, annotated GLSL is one of the formats the Smart Filter Editor can import as a block definition, and you added the required annotations correctly there. The documentation will be updated soon with more information about the supported formats, keep an eye on Babylon.js docs

Thanks!

[amoebachant]

Hey everyone, we’ve published updated documentation for Smart Filters

Please check it out and let me know if you have any feedback!

[a.dzhakhangirov]

Adding a webcam to a project is a great idea!
It would be nice to see a video on this topic.

[amoebachant]

Glad you like it! This is an easy way to visualize how your Smart Filter could apply to a video. It’s described in more detail in this new page in the docs - check it out!

Babylon.js docs