Universal scene.pick()

I have a scene with 2 areas where I need to get PickingInfo (My Studios project). The other major requirement is this needs to run either on a desktop or XR. I can easily know whether I am on desktop or headset.

The 2 uses are:

  • Detect the material being “hit” by either knowing the mesh, or the submesh in the case where the mesh has a multi-material. From there, apply all the current values of that PBR material to GUI controls, so that the material can edited. I have already the downstream code from the pickinginfo fully working.

  • Be able to get the face_id of the triangle which registered the hit on a grid mesh. From that look up the x of the hit triangle & adjust my XR implementation of the Performance Profiler, another project.

Handling a desktop is rather straight forward and documented, so I only need to really worry about XR.

In XR, I have found the WebXRControllerPointerSelection feature. There is not much other than API pages in docs, and one topic that actually instances the feature rather than using the feature manager.

First, how should I instance it for my uses? I prefer the feature manager, if possible.
Second, what / how to I add listeners, so that the appropriate callbacks are invoked?

cc @PolygonalSun

Ok, was busy Friday, but started looking at this today. As I kind of expected, while WebXRControllerPointerSelection may be implemented in the form of a WebXRAbstractFeature , it is used internally by WebXRDefaultExperience class to perform picking for the system, and not really useful at the application level.

What it does though at every frame, for each controller, a scene.pickWithRay() is called and the pickingInfo is stored with the data of the controller.

later, another observer on the controller, checks If the pickingInfo is not null, then calls either a scene.simulatePointerDown(), scene.simulatePointerUp(), or scene.simulatePointerMove() with the pickingInfo passed along.

In the case of pointerdown, scene calls_inputManager.simulatePointerDown(). The only way the PickingInfo object leaves from there is if scene.onPointerObservable.hasObservers()

Seems like the way to universally go is to throw away the desktop code of adding a canvas pointerdown listener, then calling scene.pick(scene.pointerX, scene.pointerY).

Thinking this might always works, if I knew what an EventState.mask was (just guessing).

scene.onPointerObservable.add((info: BABYLON.PointerInfo, state: BABYLON.EventState) => {
    if (state.mask === BABYLON.PointerEventTypes.POINTERDOWN) {
        info.pickInfo // do something with this