Future of styles and GUI controls?

[HiGreg]

I’ve been thinking about how to implement GUI styles and wanted to pick the brains of those who’ve already thought deeply about it. How should the style class be improved? Are there any plans for the future of styles?

Background:
I was recently noticing how difficult it is to implement a consistent button look and feel. Simultaneously, the current style class only implements font-related properties (“for now”).

I wrote a very simple “copy properties” class that saves/loads a control’s look and feel. The discussion is found here and with playground, primarily implementing ApplyStyleToFrom(toControl, fromControl). It makes me wonder “how can it be improved?”

Questions:
1. Is CSS the aspirational model or are Babylon styles their own thing?
2. How similar to CSS programmatically should styles be? (likely CSSStyleDeclaration, but what about the other elements of CSSDOM?)
3. Can and should Babylon integrate with CSS where styles declared in HTML pass down to Babylon?

I think it’s fair to say that a primary limitation is the maintenance burden on core developers and the very related preservation of backward compatibility now and into the future.

Regarding CSS, there’s a ton there! The general categories: properties, cascading and cascade layers, inheritance, selectors, specificity, values/units, sizing, styling backgrounds and borders, animation, transitions, transforms, CSS flexible box, multi-column and grid layout, stacking and block-formatting contexts, etc, etc.

If this is the chosen path, it seems to start with inline style declarations because internal and external declarations would naturally use inline style functions. And no selectors are needed for inline, though inheritance is a little bit of a slippery slope towards needing selectors.

Inline styles almost immediately gets into properties, shorthand properties, values, units, and functions (calc()?) including dynamic and relative values. Knowing where we want to end up helps greatly with the first steps. It also helps inform how deeply modifications are warranted to other Babylon objects, such as fonts, AdvancedDynamicTexture, and GUI controls generally.

I hope you don’t mind me tagging you, I think you might be interested in this conversation: @mawa @jamessimo @PatrickRyan @tricotou @RaananW @carolhmj @julien-moreau @Dok11 @kintz_09

Shout outs to @RaananW for the initial implementation and continued maintenance of styles, @mawa for his boundless enthusiasm and encouragement and this positioning statement from PatrickRyan

@PatrickRyan

the Babylon GUI system is a stand-alone system build for the engine that takes inspiration from CSS, but is not a direct re-implementation of CSS. So there will be things like rem that have not been implemented into Babylon GUI. That does not mean they will never be there if there is enough call for them to be developed. And there’s nothing stopping anyone from contributing to the Babylon GUI system to add features like this before the core team can devote resources to them.

If/when it becomes prudent, I’ll add a list here (or inline) of posts and links that have related questions, comments, ideas, and solutions that may help spur the conversation.

[HiGreg]

I’m working on a prototype to establish feasibility and performance. First step I am trying to get to is the ability to define and propagate properties to Controls. The properties in the string at first will be simple exact names of Control properties instead of CSS properties, but can include CSS units. Here’s a description of the two classes I have so far. No working example yet. I apologize for the poor grammar and capitalization.

class Value() represents a single non-composite CSS-like value. It can convert from string, recognize string vs identifier vs numeric, handle units, and percentage. Enables relative values but doesn’t insert them itself.

After processing, the following fields are available:
isIdentifier, valueString, unitString, type, multiplier, number. gdtComputedValue() calculates number * multiplier where multiplier is derived from unit conversion to “base” value (length cm, angle rad, time s, frequency hz, resolution dppx).

Once dependencies among properties (from absolute units and calculation inputs) and to Control properties (from “which style properties affect Control properties”) are determined, a topological sort can be created outside of this function to determine recalculations and Control property updates needed when any property changes occur.

class NewStyle() handles Conrol hierarchy changes and (will) track calculation and property dependencies to minimize recalculation and property updates.

There is a HACK to detect parent changes. The only option currently is to save previous parents and check every draw. One consequence of this is that styles are not available until control’s first draw after control or ascendant has been added to adt. You can work around this by assigning a style before adding it to adt hierarchy:
const style = new NewStyle(control)
though there’s no way to obtain computed values until after the first draw. A lesser alternative is to modify style within onAfterDrawObservable.addOnce() (or possibly as the last “beforeDraw” observer) and use
GetControlStyle(control) to obtain its style (including computed and used).

[mawa]

Sorry, don’t have much time this week. I will come back to it later I’m still very interested in this, sadly have other things I absolutely need to attend to … will be back
Meanwhile, have a great day

[HiGreg]

One of the things I’ve come across is that, in my prototype, the hierarchy of “styles” is expected to follow the hierarchy of controls.

Do you think this is too severe of a limitation? All controls will naturally inherit style from the AdvancedDynamicTexture’s root container. The issue will come up when you want different sets of styled controls. Each set of controls will need to be parented to a common control from which it derives its style. For controls sharing style all within a StackPanel, this is not a problem. But if you have controls all parented directly to the ADT but you need different sets of styles and place each set in their own container, then I don’t know how zIndex is applied. Are all controls in one container drawn either all above or all below controls within another container?

Ideally, each control interleaves with all others based on the individual control’s zIndex. I’m not sure if/how this would affect other styles.

This also gives me an idea, which I’ll save here: is it possible to have relative zIndex (i.e. relative to a parent’s zIndex). Not sure, yet, if CSS can have arbitrary “relative to” properties. There are built-in relative values such as parent’s width.

[mawa]

@HiGreg

Each set of controls will need to be parented to a common control from which it derives its style. For controls sharing style all within a StackPanel, this is not a problem. But if you have controls all parented directly to the ADT

Ok, my opinion only: Parenting everything to the ADT, is no ‘structure’ for me. It seems obvious that the container or control inherits. To me an ADT is not a ‘setter’ or ‘container’. It just is… the ADT.

[mawa]

@HiGreg

(i.e. relative to a parent’s zIndex). Not sure, yet, if CSS can have arbitrary “relative to” properties

Honestly, it might just be my small brain or else, my long experience in PM but I’d say, dont’ make it too complex for a start. If we look at CSS (and remembering that css took 3 decades to make it to the current ), ‘relative’ is relative to the parent. And ‘absolute’ is just non-relative. May be we should just start with this (generally). As for the zIndex, there’s all sorts of fancy building around it (obviously in this aspect we are not css). Messing around with the hierarchy might raise all sorts of issues (with the Editor and the plug-ins such as the one from @jamessimo ). It’s just a feeling more than a code-based analyses, but I think we should be careful with this part. We can still improve and come-back on it later (again, my opinion/feeling only).

[HiGreg]

Agreed. All controls inherit style from their parent but then can modify the style for themselves and their children.

The issue I’m highlighting is when you want the control hierarchy different from the style hierarchy. But maybe you’re saying no one should expect it to be different?

However, I thought a good demo would be a pyramid of buttons and when clicking on one the style would change on it and all it’s children. I can’t, currently in the prototype, set a hierarchy of styles that is different than the hierarchy of the controls. Which means I think I have to create a separate container for each button that contains all it’s children, which then each become containers for their children. All the way down. This seems like it shouldn’t be necessary, but is a consequence of style hierarchy == control hierarchy.

[mawa]

Sry, I have to leave now. I’m gonna need some time to think about it anyway Hopefully, meanwhile, some other will kick-in and feed the project with their own experience and expertise … 'Chat with you later,

[mawa]

@HiGreg

This seems like it shouldn’t be necessary, but is a consequence of style hierarchy == control hierarchy.

I believe ‘containers’ only should spread styles to the children (not controls). A control would only spread to his own ‘parts’. But that’s just ‘high-level’ and as of just now. As I said, would need more time to think about it (and would also like to hear from others).

[HiGreg]

@mawa

We can still improve and come-back on it later (again, my opinion/feeling only).

Agreed. I view my prototype to be a demonstration and enable experimentation. I expect it might be the basis of a future update to Style. I’m just putting thought on how should end up.

I already have hierarchical flowing of styles with separate “specified values” and “computed values” where computed values are passed to child controls (passed in absolute form, though still computed relative to it’s parent). I’m not yet adopting existing CSS keys, instead using the Control properties. But I do want to get a demonstration of the hierarchical flowing of styles to get a handle on performance and memory impacts.

[HiGreg]

@mawa

I believe ‘containers’ only should spread styles to the children (not controls). A control would only spread to his own ‘parts’.

At first glace, this seems like a simplifying view. However, if only containers passed styles to their children, then every composite non-container control (such as Button with image and text controls within it), would need a custom function for applying styles to its constituent parts. The current paradigm of “all controls pass styles to their children” means that every control, or any Object that has a parent and implements getDescendants will pass its style to those descendants.

Not sure this is the best consideration, but it greatly simplifies the code in that no special code is needed for any current or future control (hopefully!). I was thinking mostly about flowing font information to child controls. I’ll have to think about the case of a composite control where constituents are differently styled in position (e.g. a button centered in it’s container with an child image in the button’s upper left corner). This might be handled by specifying different values for the image positioning within the image’s style. This seems no more onorous, and perhaps less so, than assigning custom positioning values on the image when there is any change to the composite control’s size. “Less so” because you could assign a relative position on the child image and it would flow automatically. This might be especially powerful when fields allow functions (e.g. calc()).

Edit: an example that elucidates that the “simplifying assumption” is actually more complicated is in the assignment of fonts to child controls of non-container controls. If styles flowed freely through to all children, then an assignment of a font to all controls can be done with a single assignment of the font to the root style. If styles don’t flow freely through to child controls of non-container controls, then constituent TextBlock and InputText controls must be individually styled for fonts.

Ultimately, applying completely different styles to constituent controls could be defined by CSS Selectors in a style sheet. Marking this idea for future: IDEAMARKER. CSS-like selectors should accomodate Control types and parent types. (I think CSS itself already allows for that).

In addition to that, existing CSS keywords specifying an inheritance rule (inherit, initial, revert, revert-layer, and unset) can be specified by a style declaration and applied to specific properties or “all”. That said, this is getting close to “not initially implemented,” but I want to enable that in the future by having compatible definition, scoping, and code insertion points of and within class methods and properties.

A few things not initially implemented (but hopefully enabled) are Selectors, StyleSheets (only Declarations at first), full “relative to” logic, functions within fields, animations, transitions, gradients, image definitions, CSS Layer hierarchy, Layout logic different than layout inherent in current Babylon controls, and fields that aren’t BABYLON Control properties.

That said, one thing I’d like to get to after this initial implementation is StyleSheets with an @ babylon selector. Then at least the top-level babylon style can be defined in a single string or an external file. And, possibly, within a stylesheet containing CSS for non-babylon parts of the page.

[PatrickRyan]

@HiGreg, I spoke with @RaananW about what you are proposing, and we have some thoughts. First and foremost, we both agree that we need to build the next version of the core GUI system all up to replace what we currently have. This new system would need to be integrated with the editor, and we would need to deprecate the current GUI system without removing it for back-compat reasons. Where our current system takes inspiration from CSS, we definitely didn’t implement things in the same way originally. For example, I think we have a conflict between margin and padding between Babylon GUI and CSS where we use margin, but it’s really padding in CSS or vice versa, I don’t remember exactly which off the top of my head.

Because of those incompatibilities if we tried to drive CSS from the DOM down to the Babylon scene using our current system, we would have to remap all of those inconsistencies while also adding in the rest of CSS we don’t currently support. Additionally, once we include Babylon Native in the discussion, how would Native integrate any system ingesting CSS which makes the question a little more tricky to answer. On the other hand, if we expanded the styles to be more inclusive of all control properties, we would be adding to a system that needs to be overhauled from the bottom up and then continue support for it once it is deprecated for back-compat.

Something you can do right now is create an extension to the engine that enables your styles implementation and we are happy to host it on our extensions repo, so long as you are willing to maintain it as it would be a community extension. Additionally, one of the pet projects that @RannanW has is to build out a way for users to generate community extensions through NPM that can be added through simply installing them with NPM. Your system looks like a great candidate for an extension so it can be pulled in for anyone who wants to use it without placing it in core. Anything integrated in core will shift maintenance to the Babylon team for bug fixes and integration with any new features or an update once we rebuild GUI.

The big question on the horizon is when any resources can be devoted to GUI to rebuild it and I honestly have no answer for that. Our team is small and split between Babylon.js and Babylon Native. What I can say is that if rebuilding GUI is important to the community, we will find a way to fit it in. And the best way for us to show there is a big desire for the feature is to drop it in the Feature Requests section of the forum and generate a lot of votes for it. We do look at the votes for new features when we are planning out what will comprise the next Babylon release.

In the meantime, I will leave you with what I do for styling controls when writing them in code. In the GUI Editor, it’s easy to copy and paste a control, but when writing them manually, I just create a function for each style. Something like

function createMyButton(name, text) {
     let button = BABYLON.GUI.Button.CreateSimpleButton(name, text);
     button.background = "powderblue";
     button.color = "black";
     // rest of the button parameters needed, size, corner radius, 
     return button;
}

function buttonDown(button) {
     button.background = "green";
     button.color = "white";
}

function buttonUp(button) {
     button.background = "powderblue";
     button.color = "black";
}

And then create your button using that style:

let newButton = createMyButton("newButton", "Press Me");
newButton.onPointerDownObservable.add(() => {
     buttonDown(newButton);
     // all button down behaviors
}
newButton.onPointerUpObservable.add(() => {
     buttonUp(newButton);
     // all clicked behaviors
}

// add other state behaviors

This works like the copy style example you gave other than you don’t have to actually copy any styles over. You can also add specific sub-styles by creating a function to set just a subset of parameters to any control you pass to it. This could be done on a variety of controls like if you want all controls to be right aligned in one place in the ADT, you could set up an array of those controls and send them to a function that sets alignment. There are several parameters that are agnostic of control so a function could set parameters across different types of controls. This approach is similar to a style meaning you only have to write it once, but rather than just applying an id or class to the control, you would need to pass that control to the function that sets the style.

I have been using this approach for a while now and it greatly speeds up my work and greatly reduces the amount of GUI code I need to write. I wish there was an easier solution to make the GUI system better, but without a ground-up rewrite which can improve our current system and be planned to work better with CSS in general - which both @RaananW and I feel we need – there is no quick solution that should be integrated into the core engine.

[HiGreg]

Agreed on all points. Thus far I have enough to keep me going on the prototype, and it is extremely helpful to know “CSS like” is the goal.

At this stage, it’s a prototype to test feasibility and performance impacts on mirroring CSS-like functionality. Once I am able to execute basic functions, I’ll want to review what is needed for GUI Editor integration. If you can point me to some Interface or API documentation, that’d be great!

It does not, as far as I know, need any significant support from the development team nor does it need any kind of overhaul to the GUI system. I’m working within the constraints of the current GUI design and existing Controls.

There’s a lot of cruft in the code at the moment, but here’s a basic demo.

My goal is to enable inheritance on styles so that children get their “initial” style from their parent, then a modification to that inherited style can be made, live, on any node (control or container) and the style is updated for that node and its children. For example, my test code looks like:

changeStyleButton.onPointerClickObservable.add((event,data)=>{
    const styleString = "cornerRadius:"+Math.random()*15
    NewStyle.AddDeclarations(ui.rootContainer,styleString)
})

The declarations string can contain as many properties as desired and the string could be loaded from anywhere. Once Selectors are implemented, it should be possible to contain styles for multiple controls in a file. It should be the same for Native applications. It’s not dependent on general HTML or CSS support. IDEAMARKER Perhaps I can design (or discover) a simple declarative markup that defines the control structure. That way, a single file could contain an entire GUI style definition and Control style declarations.

I’m trying to keep performance high and memory usage low, with an eye toward maintability. That includes the simple statement to enable the system:

NewStyle.AttachToAdt(ui);

And a mechanism to freeze the monitoring of parent changes.

Notably, I link control property modifications from CSS properties through the function table _stylePropertyAssign, with fallback of control[pName] = computedValue which works great when the CSS property name matches the Control property name.

[HiGreg]

Babylon.js Playground

Propagation of properties works well. For the demo,

• Press color buttons excluding “change below”.
• “change top” will propogate cornerRadius and alpha changes to all, automatically.
• “change below” will add the corner radius property only on the parent of the left buttons.
• Subsequent “change top” will propagate alpha changes everywhere, but cornerRadius will change only on the random textboxes, not the left buttons.
• This is because “change below” modifies a more specific control (lower in the hierarchy than “top”), and that specificity blocks the propagation of cornerRadius to the left buttons.

[mawa]

@PatrickRyan

For example, I think we have a conflict between margin and padding between Babylon GUI and CSS where we use margin, but it’s really padding in CSS or vice versa, I don’t remember exactly which off the top of my head.

Huh? Honestly, in css, nobody does (remember) As far as I can recall, the logic and understanding of margin VS padding (out of containers) is something you just need to get acustomed to.

@PatrickRyan

I wish there was an easier solution to make the GUI system better, but without a ground-up rewrite which can improve our current system and be planned to work better with CSS in general - which both @RaananW and I feel we need

Aside from that, I can say I’m glad I brought you into this discussion (and thanks again for your dedication - I do know the BJS GUI is important to you… as it is to me). I don’t think we need a copy of css but on the other hand, css has gone through a lot and has now 3 decades experience… ignoring this experience would be foolish (yet, as I say, without the goal of actually porting DOM to GUI, which I believe would be a challenge at this point and the current resources).
Let’s just say I’m happy that, with your support and the one from the Team, people like @HiGreg or @jamessimo or @Tricotou (and humbly, myself ) will continueously push for the BJS GUI, making it better and in fine, making it a standard to use with a BJS scene

[HiGreg]

I appreciate the conversation!

Digging into CSS Layout, I’m working out how to apply it to GUI. At first I though block layout would apply most directly. But inline might apply inside a TextBlock? Then I remember there is already an HTML Texture. IDEAMARKER I’m wondering if GUI could apply Images from Textures. Just a random thought. grid layout might implemented as a generated grid Container.

I’ve also found the CSS parser used by the developer of CSS Syntax. 17k minified, it will be useful for parsing style sheets, selectors, and declarations.

I almost have enough to generate a table of planned support of CSS at the keyword level. And hopefully a development path and priority. Expect that in a week or two.

[mawa]

@HiGreg

I though block layout would apply most directly. But inline might apply inside a TextBlock?

My two-cents and ‘high-level’ only: Speaking of ‘styling’ (in the sense of class/id) to me, refers to block display. In fact, in css, I mostly use inline either where I need explicit declaration to ensure consistency accross all clients (i.e. email templates) OR as a ‘hack’ to override a class/id because I need it just once or twice and don’t want to make another style for just that.

Again, ‘high-level’, I think we should ‘exclude’ the textBlock from the first logic. Look at the TB like a ‘special component’ that is bound to have its own logic and method for ‘styling’. Indeed, originally, my suggestion for revamping the TB was also to be able to apply ‘simple’/limited ‘inline-like’ css styling. I kind of believe this is still the way to go for this particular control… but it is more of a ‘feeling’ than based on a proper analysis.

@HiGreg

I remember there is already an HTML Texture

Yes. And at first, I thought it would become the replacement for the abandoned ‘styled TB’ project. But honestly, I don’t really think so anymore. It’s a nice adding and it has its use cases but to me it doesn’t really match the requirements I (personally) expect from a styled text block. Again, my opinion only

@HiGreg

I almost have enough to generate a table of planned support of CSS at the keyword level. And hopefully a development path and priority

That sounds really great and would be a ‘must have’ base to go down this path. You’re about to become my hero I’m eager to put my eye on this one