NME custom blocks take two

Hey y’all, here’s the first draft of the second incarnation of custom blocks for NME! :sweat_smile:

It’s based on user-defined, simple json files.

So far only the features needed to port MultiplyBlock have been implemented. Saving and loading materials that use custom blocks seems to be working well also.

The next steps I think are to provide default values for all of the properties, which have been omitted mainly to facilitate testing, and to port a more complicated block, like WorleyNoise3DBlock, and implement the additional features needed for it along the way. And moooore testing.

So not done yet, but I wanted to stop and see if I’m on the right track? :face_with_monocle:

Here’s the json file for testing/reference: multiplyBlock.json

Note: the glsl source code is provided as an array of strings; the values for the settings are named after their corresponding constants (enum keys) from the API.

Here’s the CustomBlock.ts file that deserializes from the above json.

And here’s the patch: NME custom blocks second draft by BlakeOne · Pull Request #11114 · BabylonJS/Babylon.js · GitHub

(P.S. if the PR gets too far behind because it’s the weekend, let me know and I’ll make another patch against next week’s latest BabylonJS repo)

2 Likes

I will let the NME sorcerers @Deltakosh and @Evgeni_Popov have a look at this one

1 Like

Here’s the material that I’ve been using for testing, it just multiplies two color input blocks together, one is purple(255,0,255) and the other is gray(128,128,128) so we can verify that the mesh is the same dark purple color when using either the custom multiply block or the built in multiply block.

this is super great! Please add all the links in the issue so we can validate and merge :slight_smile:

2 Likes

@Deltakosh, @Evgeni_Popov: Here’s the new patch :slight_smile:

3 Likes

@Evgeni_Popov, maybe something like this for the json? Where only the last function in the functions array is called automatically, and if the types are omitted from the parameter list of the last function, then they’re added automatically based on the actual types used at runtime. And as you said, I think the order of the inputs and outputs in the array of parameter settings would need to match the order of the function’s parameters. Then I could use the first line of the function to derive the function’s name and parameter names, used for calling it automatically, and parameter types if present.

Edit, well I suppose the inputs and outputs could be in the same array of “parameters” now since I’m using “out” from the function params to tell them apart, then just match the order of the parameters array to the order of the last function’s parameters.

What about just always setting the output’s type to the type of the first input when the output’s type isn’t specified in the function paramaters? Maybe then all we would need is an array of displayNames, rather than an array of settings? Not sure how universal it should be, in terms of options for the type detecting?

I would see something like that:

{
    "name": "Multiply",
    "comments": "Multiplies the left and right inputs of the same type together",
    "target": "Neutral",
    "inParameters": [
        {
            "displayName": "left",
            "type": "autodetect"
        },
        {
            "displayName": "right",
            "type": "autodetect"
        }
    ],
    "outParameters": [
        {
            "displayName": "output",
            "type": "BasedOnInput",
            "typeFromInput": "left"
        }
    ],
    "inLinkedConnectionTypes" : [
        {
            "input1": "left",
            "input2": "right",
            "looseCoupling": false
        }
    ],
    "functionName": "multiply",
    "code": [
        "TYPE_result myHelper(l: TYPE_left, r: TYPE_right) { return l * r; }",
        "void multiply(left: TYPE_left, right: TYPE_right, out result: TYPE_result) {",
        "   result = myHelper(left, right)",
        "}"
    ]    
}

I think it’s better to separate the in from the out parameters, it may come in handy later on when we have a UI to build a custom block (having to write a .json file by hand should be a temporary solution, as a starting point).

In the glsl code, TYPE_XXX allows to retrieve the type of the input/output named XXX.

With this .json, you should be able to generate the same block than the existing MultiplyBlock:

  • the "type": "BasedOnInput" and "typeFromInput": "left" will let you build the this._outputs[0]._typeConnectionSource = this._inputs[0]; call (found in the MultipltyBlock constructor)
  • the inLinkedConnectionTypes entries will let you build the this._linkConnectionTypes(0, 1); call (found in the constructor)
  • it’s easier to simply ask the user the name of the function they want to create/read instead of parsing the glsl code. That’s why there’s a functionName entry in the .json file
1 Like

I think it’s better to have the user describes precisely the types of their inputs/outputs (note that it’s possible to define several outputs to a function), as in Unity. It’s not really more work for them, and it’s easier for us.

1 Like

After kicking this around a bit, I think that the fundamental issue is that setting associatedVariableName doesn’t work. The API documentation for associatedVariableName says “Gets or sets the associated variable name in the shader”. If this were true, I could simply set associatedVariableName to the options’ param name and remove the {} escape sequences from the GLSL code (the original issue) and remove all of the name-replacing code as well. Then the glsl code in the json file would be much simpler, as it originally was.

So it makes more sense to me to put work into fixing NodeMaterialConnectionPoint’s associatedVariableName implementation, which would make the glsl simpler by removing the escape {} from the param names, instead of putting work into making the glsl code more convoluted by adding the wrapper to it.

And for adding global code (outside of main), my plan in porting worleyNoise3DBlock, which uses this feature, was to simply have an optional property that allows another array of code to be emitted before the main function (by simply calling _emitFunction like worleyNoise3DBlock does).

So this is my thinking on the issue at this point. :thinking:

You don’t need to have {} around the parameter names. The variables you need to pass to the function call are all the input.associatedVariableName and all the output.associatedVariableName: those are generated by the system and you don’t need to tamper with.

I think it’s easier to see what’s going on with real code. I modified the CustomBlock file to make it work with this .json:

{
    "name": "Multiply",
    "comments": "Multiplies the left and right inputs of the same type together",
    "target": "Neutral",
    "inParameters": [
        {
            "name": "left",
            "type": "AutoDetect"
        },
        {
            "name": "right",
            "type": "AutoDetect"
        }
    ],
    "outParameters": [
        {
            "name": "output",
            "type": "BasedOnInput",
            "typeFromInput": "left"
        }
    ],
    "inLinkedConnectionTypes" : [
        {
            "input1": "left",
            "input2": "right",
            "looseCoupling": false
        }
    ],
    "functionName": "multiply_{TYPE_left}",
    "code": [
        "{TYPE_output} myHelper_{TYPE_left}({TYPE_left} l, {TYPE_right} r) { return l * r; }",
        "void multiply_{TYPE_left}({TYPE_left} l, {TYPE_right} r, out {TYPE_output} result) {",
        "   result = myHelper_{TYPE_left}(l, r);",
        "}"
    ]    
}

CustomBlock class:

import { NodeMaterialBlock } from '../nodeMaterialBlock';
import { NodeMaterialBlockConnectionPointTypes } from '../Enums/nodeMaterialBlockConnectionPointTypes';
import { NodeMaterialBuildState } from '../nodeMaterialBuildState';
import { NodeMaterialBlockTargets } from '../Enums/nodeMaterialBlockTargets';
import { _TypeStore } from '../../../Misc/typeStore';
import { Scene } from '../../../scene';
import { NodeMaterialConnectionPoint } from "..";
import { Nullable } from "../../../types";
/**
 * Custom block created from user-defined json
 */
export class CustomBlock extends NodeMaterialBlock {
    private _compilationString: string;
    private _options: any;

    /**
     * Creates a new CustomBlock
     * @param options defines the options used to create the block
     */
    public constructor(options: any) {
        super("emptyCustomBlock");

        if (options) {
            this._deserializeCustomBlock(options);
        }
    }

    /**
     * Gets the current class name
     * @returns the class name
     */
    public getClassName() {
        return "CustomBlock";
    }

    /**
     * Builds the block's compilaton string
     * @returns the current block
     */
    protected _buildBlock(state: NodeMaterialBuildState) {
        super._buildBlock(state);

        let code = this._compilationString;

        let functionName = this._options.functionName;

        // Replace the TYPE_XXX placeholders (if any)
        this._inputs.forEach((input) => {
            const rexp = new RegExp("\\{TYPE_" + input.name + "\\}", "gm");
            const type = state._getGLType(input.type);
            code = code.replace(rexp, type);
            functionName = functionName.replace(rexp, type);
        });
        this._outputs.forEach((output) => {
            const rexp = new RegExp("\\{TYPE_" + output.name + "\\}", "gm");
            const type = state._getGLType(output.type);
            code = code.replace(rexp, type);
            functionName = functionName.replace(rexp, type);
        });

        state._emitFunction(functionName, code, "");

        // Declare the output variables
        this._outputs.forEach((output) => {
            state.compilationString += this._declareOutput(output, state) + ";\r\n";
        });

        // Generate the function call
        state.compilationString += functionName + "(";

        let hasComma = false;
        this._inputs.forEach((input, index) => {
            if (index > 0) {
                state.compilationString += ", ";
                hasComma = true;
            }
            state.compilationString += input.associatedVariableName;
        });

        this._outputs.forEach((output, index) => {
            if (index > 0 || hasComma) {
                state.compilationString += ", ";
            }
            state.compilationString += output.associatedVariableName;
        });

        state.compilationString += ");\r\n";

        return this;
    }

    /**
     * Serializes this block in a JSON representation
     * @returns the serialized block object
     */
    public serialize(): any {
        let serializationObject = super.serialize();

        serializationObject.options = this._options;

        return serializationObject;
    }

    /**
     * Deserializes this block from a JSON representation
     * @hidden
     */
    public _deserialize(serializationObject: any, scene: Scene, rootUrl: string) {
        this._deserializeCustomBlock(serializationObject.options);

        super._deserialize(serializationObject, scene, rootUrl);
    }

    /**
     * Deserializes this block from a user-defined JSON representation
     * @param options defines the options used to create the block
     */
    private _deserializeCustomBlock(options: any) {
        this._options = options;
        this._compilationString = options.code.join("\r\n") + "\r\n";
        this.name = options.name;
        this.target = (<any> NodeMaterialBlockTargets)[options.target];

        options.inParameters?.forEach((input: any) => {
            const type = (<any> NodeMaterialBlockConnectionPointTypes)[input.type];
            this.registerInput(input.name, type);
        });

        options.outParameters?.forEach((output: any, index: number) => {
            this.registerOutput(output.name, (<any> NodeMaterialBlockConnectionPointTypes)[output.type]);

            if (output.type === "BasedOnInput") {
                this._outputs[index]._typeConnectionSource = this.findInputByName(output.typeFromInput)![0];
            }
        });

        options.inLinkedConnectionTypes?.forEach((connection: any, index: number) => {
            this._linkConnectionTypes(this.findInputByName(connection.input1)![1], this.findInputByName(connection.input2)![1]);
        });
    }

    /**
     * Finds the desired input by name
     * @param name defines the name to search for
     * @returns the input if found, otherwise returns null
     */
    public findInputByName(name: string): Nullable<[NodeMaterialConnectionPoint, number]> {
        if (!name) {
            return null;
        }

        for (var i = 0; i < this._inputs.length; i++) {
            if (this._inputs[i].name === name) {
                return [this._inputs[i], i];
            }
        }

        return null;
    }
}

_TypeStore.RegisteredTypes["BABYLON.CustomBlock"] = CustomBlock;

With those modifications, this NM:


compiles to:

uniform vec4 u_color;

vec4 myHelper_vec4(vec4 l, vec4 r) { return l * r; }
void multiply_vec4(vec4 l, vec4 r, out vec4 result) {
   result = myHelper_vec4(l, r);
}

void main(void) {
    //Multiply
    vec4 output2;
    multiply_vec4(u_color, u_color, output2);
}

(I have only kept the relevant bits)

If adding another multiply block that is using a different type from vec4:


which compiles to (relevant bits only):

uniform vec4 u_color;
uniform float u_Float;

vec4 myHelper_vec4(vec4 l, vec4 r) { return l * r; }
void multiply_vec4(vec4 l, vec4 r, out vec4 result) {
   result = myHelper_vec4(l, r);
}

float myHelper_float(float l, float r) { return l * r; }
void multiply_float(float l, float r, out float result) {
   result = myHelper_float(l, r);
}

void main(void) {
    //Multiply
    vec4 output2;
    multiply_vec4(u_color, u_color, output2);

    //Multiply
    float output3;
    multiply_float(u_Float, u_Float, output3);
}

This example is a bit complicated by the fact we allow any type for the inputs, so we need to parameterize the glsl code with this type.

Regarding the worley shader, you should have no problems to make it work with this CustomBlock.

2 Likes

Yes I had the feeling I wouldn’t fully understand you implementation idea until I saw your implementation, LOL. And you’ve implemented the missing features (additional I/O configurations and global functions) in one fell swoop it looks like! :beers: Okay, well then I’ll resubmit the patch with your updated customBlock class and an updated what’s new. :+1:

@Popov72 it looks your change of the import for NodeMaterialConnectionPoint to import from “…” is causing a validation error. I tried fixing it but I think you’ll have a much easier time with it because I don’t understand what you’re doing there (I’m still newbie). :slightly_smiling_face:

Here's the validation error:
[19:09:29] Line 7 Imports .. needs to be full path (not from directory) for tree shaking. /home/vsts/work/1/s/src/Materials/Node/Blocks/customBlock.ts

Yes, import { NodeMaterialConnectionPoint } from ".."; is wrong, it’s this damn VSC which is adding this automatically (I need to disable this!). It should be import { NodeMaterialConnectionPoint } from "../nodeMaterialConnectionPoint";

1 Like

Okay, It’s fixed and resbumitting the patch now :+1:

Hurray, it’s all done validating successfully! Thanks for helping bring it home @Evgeni_Popov :beers: Here’s the completed patch for custom blocks: :point_down: :slightly_smiling_face:

3 Likes