Use Playwright for testing - id for 3d Objects?

Questions
1

Hi there,

i am currently setting up a test environment for a plattform wich also uses Babylon (React).
Is there any chance to query if certain elements are loaded within the babylon environment?
so for example check if there is a mesh named xyz in the babylonjs scene from outide the environment?

I would like to have playwright check certain meshes without the need of visual testing.

Thanks a lot!

2

I think i might have found a way: couldn’t i use: Babylon.js docs
to figure out if Objects in scene are present or not?

3

what we do in our environment is store the scene as a global variable and access the scene and its getXXXById functions. so we can get lights, meshes, cameras and so on.

4

Also sounds promissing! Can you give an example how that would look for you?

Thanks a lot!

5

everything is opensourced :slight_smile:

6

Hi, thanks a lot, if i understand the code correctly, you initialize the scene before rendering? Would this also be possible to get to the scene objects when the babylon environment already exists? So would there be a way to integrate it to my existing project?

I think i am not fully understanding whats happening here :confused:

7

Treat the global context object (i.e. globalThis, or window) as a way to store the containers you need to find the objects you need. This is wonderful for testing, not too recommended for production.
We initialize Engine, scene, and only then are we able to start rendering. This is the general way babylon is working. You can, of course, run a render loop without a scene object, but what are you actually rendering?

8

there are a lot of questions in my head. I think without an easy to understand example i will not see how you access the data from the playwright test.

Treat the global context object (i.e. globalThis , or window ) as a way to store the containers

I just dont know what you mean by that :confused:

In the meantime i did follow the other path to use the accessability class and failed too. How is that meant to be used? This is the minimal example i was trying to work with:

import {  Engine, Scene, useScene } from 'react-babylonjs'
import {  Vector3 } from '@babylonjs/core/Maths/math.vector'
import Accessibility from '@babylonjs/accessibility'


export const BabylonEnv = () => {
  
  const scene = useScene()
  if (scene) {
    Accessibility.HTMLTwinRenderer.Render(scene)
  }
 
  return (
    <div style={{ flex: 1, display: 'flex' }}>
      <Engine
   
      >
        <Scene>
          <freeCamera
            name="camera1"
            position={new Vector3(-10, 5, 0)}
            setTarget={[Vector3.Zero()]}
          />
          <hemisphericLight
            name="light1"
            intensity={0.7}
            direction={Vector3.Up()}
          />
          <box
            name={'box'}
            size={2}
            position={new Vector3(1, 1, 1)}
             accessibilityTag={{
               description: `A box`,
             }}
          ></box>
        </Scene>
      </Engine>
    </div>
  )
}

but the property accessibilityTag does not exist.

9

I can see a couple of things that may be an issue with your example. Let’s start with some knowns.

  1. (from HTMLTwinRenderer on Babylon.js docs) This will generate a <div id="accessibility-host"> HTML element right after the Babylon.js scene’s canvas element (this will be relevant in your test assertions).
  2. useScene will not work where you have it in the graph of the JSX. That’s because a React renderer is being used and you would need to bridge the context. I have only bridged from outside to inside and not the other way around. I would put that in a component and put it inside the <Scene>...</Scene>. Then it will work as expected. More details on context bridge here, but I don’t think you will need it: brianzinn.github.io/react-babylonjs/examples/basic/context-bridge/. More details on placement here: brianzinn.github.io/react-babylonjs/examples/hooks/scene-engine/

You can optionally create a component like this - either way you will perhaps want to guard it’s not called multiple times via multiple renders:

const Accessibility = () => {
  const scene = useScene();

  if (scene /* && ref.?? */) {
    Accessibility.HTMLTwinRender.Render(scene);
  }
  return null;
}

export default Accessibility;

I think that will create the accessability tag for you when placed correctly. You may need to trigger a re-render or put that inside the Scene component itself, but definitely inside the <Engine>...</Engine>.

ie:

import Accessibility from '../Accessibility';
return (<Engine ..>
  <Scene ...>
     <Accessibility />
  </Scene>
</Engine>);
10

If you want to consider using NullEngine. You may get some mileage from these kinds of tests:
react-babylonjs/packages/react-babylonjs/test/render.spec.tsx at master · brianzinn/react-babylonjs (github.com)

I have to confess those tests are the most elegant or thorough. As a TDD person there is really low coverage on that project. There is, however, an example there for rendering a mesh and retrieving that mesh/camera/etc. and asserting property values. I don’t want to discourage Accessibility/A11y, but would myself rather test against the 3D scene for accuracy. You could even test both ways!

I didn’t actually know that module existed. Unrelated, but I was contemplating 4 years ago dual-rendering for screen readers when I wrote this post processor for a11y ( brianzinn/post-process-a11y: PostProcess shader to test visual accessibility (github.com)). Nice to see that screen reader part as a supported product.

11

Hi there, thanks for the help!

Doing this really creates the Accessibility tag in the dom:

const AccessibilityTag = ({ children }: { children: ReactNode }) => {
  const scene = useScene();
  if (scene /* && ref.?? */) {
    HTMLTwinRenderer.Render(scene);
  }
  return <>{children}</>;
}

export const BabylonEnv = () => {
  return (
    <div style={{ flex: 1, display: 'flex' }}>
      <Engine
   
      >
        <Scene>
          <AccessibilityTag>
          <freeCamera
            name="camera1"
            position={new Vector3(-10, 5, 0)}
            setTarget={[Vector3.Zero()]}
          />
          <hemisphericLight
            name="light1"
            intensity={0.7}
            direction={Vector3.Up()}
          />
          <box
            name={'box'}
            size={2}
            position={new Vector3(1, 1, 1)}
            accessibilityTag={{
              description: `A box`,
            }}
          ></box>
        </AccessibilityTag>
        </Scene>
      </Engine>
    </div>
  )
}

but how would i get the tag onto an object?

this does not work:

          <box
            name={'box'}
            size={2}
            position={new Vector3(1, 1, 1)}
            accessibilityTag={{
              description: `A box`,
            }}
          ></box>

Thanks a lot for your patience!!

12

i think i got a little bit further with this too:

this when the scene mounts:

        <Scene
                  onSceneMount={(event) => {
                      window.scene = event.scene
                  }}
        >

and a copy of your Babylon.js/packages/tools/testTools/src/window.d.ts at master · BabylonJS/Babylon.js · GitHub

makes it work:

image
image1324Ă—402 21.2 KB

now i need to find a good way to use playwright to check for everything…

thanks a lot to you too for your help!

EDIT: you considered this in a production environment as not a good idea. What are the reasons?

1 Like
13

I realize that question is for @RaananW . I can answer from my perspective - it’s called polluting global scope. Essentially, it won’t garbage collect and hopefully no other code clobbers that variable. Also, it won’t work for multiple scenes at the same time, etc. So, where you attach the scene to the window - hopefully that’s just in your test code.

On the other hand, if you are making progress maybe it’s better for you to continue as-is with something that works and then revisit later possibly as some kind of technical debt.

1 Like
14

great for testing! if it fits your use-case, of course :slight_smile:
It really depends how you configure playwright, what code you are passing it it etc’. @brianzinn is 100% correct with everything he wrote.

15

Yes, you are absolutely right. That doesn’t work - and actually it should! It’s something missing in the React renderer. Support for IAccessibilityTag. I will add that to the library. In the meantime, you can set it yourself manually like here:

image
image679Ă—655 18.9 KB

App.tsx - nodebox - CodeSandbox

Code to set accessibilityTag for now (this is only set on init, so never updated):

<box
      ...
      onCreated={(instance) =>
        (instance.accessibilityTag = {
          description: `Box: color:${props.color.toHexString()} rotating on: ${
            props.rotationAxis
          }`
        })
      }
    >
      <standardMaterial
        ...
        diffuseColor={props.color}
      />
    </box>

I should have time next week. It’s quite easy to add that support. The reason it’s not supported is because only known types are supported - this is what gives the renderer ability to support special handling for known types like Color/Vector3 (allowing integration to third party libraries like chroma-js or react-spring, etc.). I am not aware of any other renderers that support this.
react-babylonjs/packages/storybook/stories/babylonjs/Integrations/chromaJS.stories.js at master · brianzinn/react-babylonjs (github.com). Those samples were all removed on the new docs, which link to codesandbox directly to generate the equivalent of “Playgrounds” that are used for imperative code here in the forum. Just trying to reduce the dependencies there. HTH.

Adding broader support to treat unknown like any may make sense here. I’ll think about it. Cheers

1 Like