Visual Scripting/Graph Editor Feedback (v4.0.0-beta.4)

As per @julien-moreau’s request, I’ve compiled some feedback on the Visual Scripting/Graph Editor in the Babylon Editor. (I’ve included both names just for future researching’s sake)

Starting off with the positives.

Pros

Since this is a visual scripting interface I wanted to point out that I am REALLY enjoying the visual aspect of this editor.

  • Being able to see which part nodes are currently active is awesome!
  • The banner turning red or green based on it having all the inputs it needs to function is very helpful
  • I like having the ability to categorize the nodes by shape, color, and with groups. This will be invaluable for larger graphs.
  • Having all the components necessary for testing (Preview, Logs, Inspector, etc.) in the Editor window really speeds up the process of creating graphs.
  • I like that inputs will not connect to an output if it is the wrong type (or at least that’s what it appears is happening).
  • I like that there is a large collection of nodes already that can be researched either through the List of Available Nodes, or by researching the terms on the node.

image image

Next up, some unexpected behavior.

Potential Bugs
  • Clicking “Load From…” in the File menu seems to act as a Save button instead. No other window pops up to select a file, and the “Saved” message appears at the top right.

image image

  • I might have been using it wrong, but the Multiply Node didn’t give me expected results.


Whichever number is plugged into the “a” input gets multiplied by itself, rather than the number in the “b” input.

  • Clicking on the arrows to set values in nodes updates the value in the Inspector; however, typing the value on the node does not.
  • Inversely, changing a value in the Inspector does not update the value on the node.
  • I’m not entirely certain how the Debugger works, but after adding one to my Graph, the Call Stack window still didn’t activate.

Lastly, here are a couple of features/nodes I think would be helpful.

Feature Requests
  • Undo functionality
  • Scale Mesh Node: I could only find scaling in the Mesh Transform node, but since Translate and Rotate have dedicated meshes, I figured Scale should as well.

  • A tween/interpolation node where an action happens over time. (Move to this position over 5 seconds, fade in this sound over 2 seconds, etc.)
  • The Time node from the docs sounds useful.

image

  • Easing functions similar to these
  • Ability to have two different inputs trigger one node. Currently, they override each other, and I wasn’t able to connect the Or node since it didn’t have the right output to start the next node.
  • Nodes pertaining to materials could be useful. Change material, material parameter, or texture input (I know this is getting a bit close to the Node Material Editor realm)
  • Similar to how you can press Ctrl + Space in the Playground and get common code snippets, a Graph Editor equivalent could be really helpful for quickly getting up and running. Even if it was the same list, or a simplified version of the Playground list, I think that would be awesome.

That’s all I have for now!
As always, thanks for the awesome work you do on improving the Editor.

3 Likes

Excellent writeup from @tbunker as usual.

I have a followup question for @julien-moreau related to the Feature Requests on node types; I first mentioned it as a reply in the repository.

Is there a guide for contributing and testing nodes/actions to the Editor’s visual scripting engine? If not, would you be willing to make a quick-and-dirty one?

The use case is that we are porting Sumerian Behaviors (visual scripts implemented as state machines) to Babylon, and some of those SMs required actions particular to the Sumerian engine. I am confident that the underlying functionality can be ported to Babylon.

Hey @tbunker

Thanks a lot for ALL these feedbacks, bug reports and feature requests!
I’m focusing on the bug fixes right now and I’m adding the feature requests as issues on github :slight_smile:

I’m releasing a new beta with the bug fixes ASAP

I already fixed the math nodes (multiply, add, etc.), I’m dumb, I always computed the first input of these nodes. So “a * a”, “a + a”, etc. -_-

2 Likes

@julien-moreau I replied with a first draft at a custom node for use with the open-source Sumerian Host system. Let me know if I’m on the right track! I was able to build it and import my node, but I was unable to test the functionality.

Also, I’m not sure if using properties/widgets is a bad practice instead of having input nodes.

Thanks!

Hey @jkeys just saw your post !
As a quick reply, I highly suggest to use input nodes instead of widgets/properties to get nodes references.

So your node would look more like:
‘${node.code}.metadata.Emote.play()’

Anyway, I can see that TransformNode node is missing and I’m adding it! (Today only Light, Camera and Mesh exist). I’m adding it ASAP!

1 Like

@julien-moreau I saw, that’s fantastic :slight_smile:

We need two more nodes: interpolation/tween (already registered with you as a request by @tbunker), and add listener.

We use a simple PubSub implementation with regex support for listening and dispatching events between modules and components within the application.

There are a million of these PubSub libraries though, so rather than support a node for that, I was thinking there could be two new nodes:

  • dispatch (synthetic) event
  • listen to (synthetic) event

And this would use the window.addEventListener and elem.dispatchEvent (src) API. Because that is the only PubSub API that will be stable across browsers (besides maybe postMessage but that’s designed more for interprocess communication as opposed to inner-process communication).

Then on our side, for the events that we want to react to editor-generated scripting code, we could add wrappers around our Hub listeners with addEventListener and vice-versa (wrap Hub.dispatch around elem.dispatchEvent). And for anyone creating a fresh application they could just use the native pubsub API from the start to prevent needing a compatibility layer.

The generated code for the listener event would basically look like:

window.addEventListener('myCustomEvent', (evt) => {
   ${theRestOfTheCode};
});

And the generated event for the dispatch/emit event would look like:

const eventId = new CustomEvent('myCustomEvent', { detail: myCustomProperty });

What do you think?

Hey @jkeys and @tbunker

The “Interpolation Animation” node already exists, can you give me more informations about what you are looking for by “interpolation/tween”? Anyway, I just added the support of easing functions for the interpolation node:

For the window events, I just 100% agree! Just added as a try and looks like it works great:

That means all events will be sent on “window” (window.dispatchEvent(…)) and listeners will be added to window (window.addEventListener).

You can pull “release/4.0.0” right now if you’d like to take a look and share some feedbacks :slight_smile:

2 Likes

Excellent !! No problems, I have written 0 documentation about the graph editor of the V4, that means everything is my fault :slight_smile:

I’m releasing ASAP with the support of easing functions and a lot of bug fixes for the visual scripts editor. I also fixed the call stack panel which was clearly buggy.

I also fixed the Debugger node to pause the graph when it is running (it was bugged)

Also, i added a way to run ALL the graphs of the project when clicking on « play » in the visual script editor: that would allow to test the catch of events sent by other visual scripts.

2 Likes

LOL. don’t be too hard on you

2 Likes

@julien-moreau excellent work :smiley: The code is pretty easy to read and understand without documentation tbh. Although can you explain the {{generated__body}} magic? Where is that variable coming from, what parser understands the {{}} syntax…how did you achieve this?!

Not sure how @tbunker feels but I think we’ve pretty much got all the nodes we need at this point.

Woops, I accidentally double-clicked an ellipses and deleted the comment before this reply. Then couldn’t undelete it when I clicked that button. :man_shrugging:

Anyway, summary of that comment:

  • Interpolation Animation works as a tween node
  • Debug node has a record button that you click in order to activate it

Hey @jkeys

There is no really any magic trick, when a node represents a function which has a callback (like the TimeoutEvent node), its output type is set to “CallbackFunction” and the {{generated_body}} is replaced by a shy but performant .replace on the final code string of the node :slight_smile:

The code generator (still WIP and suggest to change) of graphs recursively visits the nodes (finding straight forward paths etc.), generates the code and then rides up to finally replace what is needed to be replaced.

@tbunker goooooood!!
For the debugger node I can clarify by saying: all triggered nodes (nodes with green title bar by default) have the “record” button (the red circle) that can be toggled. If you enable the red button (which is called a “break point”) each time the node is being executed the graph will pause on that node before it is executed. The debugger node simply forces that break point and generates a “debugger;” instruction in the generated code. So developers like @jkeys can pause the execution of code and see what happens. This can help to debug graphs if something went wrong with the Graph Editor.

In a perfect world, the debugger node should not exist and the Graph Editor would be perfect but I prefer to keep it right now as it is still a beta :slight_smile:

As a bonus, I just added a new node in the Graph Editor that allows to send messages to an object in scene that has an attached script. The attached script just has to implement:

...
    /**
     * Called each frame.
     */
    public onUpdate(): void {
        // ...
    }

    /**
     * Called on a message has been received and sent from a graph.
     * @param message defines the name of the message sent from the graph.
     * @param data defines the data sent in the message.
     * @param sender defines the reference to the graph class that sent the message.
     */
    public onMessage(name: string, data: any, sender: any): void {
        console.log("Received message with data:", data, "from graph", sender);
    }
...
2 Likes

Ah that makes sense. And I assume CodeGenerationOutputType.Function generates a codeblock but not inside a function?

I can’t find it =/ mind sharing the path to the code generator or a github URL to the file?

Beautiful, this will be useful for writing advanced logic on objects and invoking it with a single message. An example on the syntax of that would be helpful.

Also, as you noted, with window events we no longer need our custom emote and gesture nodes. Brilliant. :smiley:

@jkeys sure I’ll share you exemples of messages :slight_smile:

You can find the main code of the generator here: Editor/generate.ts at release/4.0.0 · BabylonJS/Editor · GitHub It still requires a polish phase.

You are right, all Function nodes define the code doing a action (assign a value to a property, call a function of an object, etc.). Practically, a function node is an action that can be represented in a single line of code.

1 Like

@jkeys @tbunker as an example, I took the “FPS Using Graphs” example and my camera has a script attached to it:

The code of the script is:

import { FreeCamera, InstancedMesh } from "@babylonjs/core";

import { visibleInInspector } from "../tools";

export default class PlayerCamera extends FreeCamera {
    @visibleInInspector("KeyMap", "Forward Key", "z".charCodeAt(0))
    private _forwardKey: number;

    @visibleInInspector("KeyMap", "Backward Key", "s".charCodeAt(0))
    private _backwardKey: number;

    @visibleInInspector("KeyMap", "Strafe Left Key", "q".charCodeAt(0))
    private _strafeLeftKey: number;

    @visibleInInspector("KeyMap", "Strafe Right Key", "d".charCodeAt(0))
    private _strafeRightKey: number;

    /**
     * Override constructor.
     * @warn do not fill.
     */
    // @ts-ignore ignoring the super call as we don't want to re-init
    private constructor() { }

    /**
     * Called on the scene starts.
     */
    public onStart(): void {
        // For the example, let's configure the keys of the camera using the @visibleInInspector decorator.
        this.keysUp = [this._forwardKey];
        this.keysDown = [this._backwardKey];
        this.keysLeft = [this._strafeLeftKey];
        this.keysRight = [this._strafeRightKey];
    }

    /**
     * Called each frame.
     */
    public onUpdate(): void {
        // Nothing to do now...
    }

    /**
     * Called on a message has been received and sent from a graph.
     * @param message defines the name of the message sent from the graph.
     * @param data defines the data sent in the message.
     * @param sender defines the reference to the graph class that sent the message.
     */
    public onMessage(name: string, data: any, sender: any): void {
        switch (name) {
            case "launch-ball":
                console.log("Just created a ball instance", data as InstancedMesh, "named: ", data.name);
                break;
        }
    }
}

The method “.onMessage” will be called each time a graph sends a message to the camera.
As a graph example, I get the camera reference and then send a message to node giving the name of the message and some data (here the instanced mesh created in the graph):

Finally, when testing the game, my code is called on the logs appear:

2 Likes

@tbunker @jkeys v4.0.0-beta.5 has just been released: BabylonJS Editor v4.0.0 beta · BabylonJS/Editor Wiki · GitHub :slight_smile:

2 Likes