How to pick a mesh or click a mesh in WebXR?

Hi,
I’m trying to change the PhotoDome texture on click of sphere meshes each loaded with 360 image textures. Please let me know how this can be done using WebXR. The pointer events (pick, down etc.,) work in browser but not when switched to VR mode (pointer move event only works).
Thanks…

Initialized XR using,

xrHelper = await scene.createDefaultXRExperienceAsync();

Sphere is below,

var sphere1 = BABYLON.MeshBuilder.CreateSphere("vrImgHolder", { diameter: 5, diameterX: 5 }, scene);
  sphere1.position.y = 2;
  sphere1.position.x = 0;
  sphere1.position.z = 0.5;
  var _material = new BABYLON.StandardMaterial("sphere material", scene);
  var _texture = new BABYLON.Texture("img1.jpg", scene);
  _texture.vScale = -1;
  _material.ambientTexture = _texture;
  _material.emissiveColor = new BABYLON.Color3(1, 1, 1);
  sphere1.material = _material;
  sphere1.isPickable = true;

The pointer event,

 scene.onPointerObservable.add((pointerInfo) => {
         switch (pointerInfo.type) {
             case BABYLON.PointerEventTypes.POINTERMOVE:
                 var pickResult = scene.pick(scene.pointerX, scene.pointerY);
                 if (pickResult.hit) {
                     var pickedMesh = pickResult.pickedMesh;
                     if (pickedMesh) {
                        dome.photoTexture = new BABYLON.Texture(pickedMesh.material.ambientTexture, scene, false, true);
                          }
                 }
              break;
             case BABYLON.PointerEventTypes.POINTERPICK:
                 break;
            }
                
        });

Hi @SenthilSelvan and welcome to the forum. This may help Use the WebVR experience helper - Babylon.js Documentation if not post to say so and see if someone with more VR experience can help.

Thanks for the reply, @JohnK. But I’m trying to do that using WebXR, as WebVR is deprecated. Please advise if the same can be used along in WebXR.

Sorry for that, I am not a VR user and my saying yes would only be a guess. Let’s ping an expert @RaananW

Anything you can do in babylon.js will work in WebXR as well. With a bit of changes sometimes, but should work.

Want to create a playground so I can understand what exactly you are trying to achieve?

@RaananW, Thanks for the reply. Here 's the playground…

https://www.babylonjs-playground.com/#542YVK#5

you can see the click on sphere works in browser and in mobile. But how to get this work in VR / XR gaze or a controller input?

This is because you are using the mouse position for picking. Your pointer event already includes a picking info object, you sure use it instead:

https://www.babylonjs-playground.com/#542YVK#6

This works in XR

Thats great… It works. But in playground only. In my mobile, the playground stuff you gave works right with a circular cursor. But if i use the same code from my localhost, it show some triangular mesh like picker instead of the circular cursor and it does not do anything!?
Please throw some light on this.

This is the gaze support.

The ray’s display should have been fixed by now (should technically be invisible until needed), I will check that soon. Having said that - there is no gaze support just yet for devices without user input (like the classic cardboard) just yet. This ticket will include this feature, when implemented - [XR] changes to PointerSelection behavior · Issue #7974 · BabylonJS/Babylon.js · GitHub . We support gaze with devices like the gearvr, which has a few buttons on its side.

Thanks for the reply… If gaze is not supported, how that works in playground in my mobile? Are gaze support and mouse position picking different? For now, I need to have the same circular cursor mesh picking which seems to work in the playground. Please advice…

What version are you using locally?

I remembered I fixed the ray, but I just realize I already implemented the gaze selection as well… I love it when I don’t remember what I did or didn’t do.

It feels like you are using an older version. Can you try using the latest 4.2 alpha?

Tried using both from cdn and Babylon.js-4.2.0-alpha.18. The references are below,

<script src="https://some.local.ip/xyz/dist/babylon.js"></script>
<script src="https://some.local.ip/xyz/dist/loaders/babylonjs.loaders.min.js"></script>
<script src="https://code.jquery.com/pep/0.4.3/pep.js"></script>

Does this need any other reference?

no idea where this points to https://some.local.ip/xyz/dist/babylon.js :slight_smile:

Have you tried our preview cdn? or the regular CDN?

It feels like you are using an earlier version, especially if it does work on the playground. Make sure you don’t reference any older version of babylon, and maybe share a proejct if you can.

““https://some.local.ip/xyz/dist/babylon.js””
its from the localhost…
just noticed in console “Babylon.js v4.1.0 - WebGL2”. This shown for both CDN (https://cdn.babylonjs.com/babylon.js) and for localhost which i downloaded the Babylon.js-4.2.0-alpha.18 zip file and included.

Finally got it working on preview cdn…(https://preview.babylonjs.com/babylon.js)…

Thanks for the support

2 Likes