GPU Multi Picking issues

Questions
1

I refer to the gpu multiPickAsync in the document to do the occlusion detection of the annotation. Every time I click to add an annotation, the pickList is updated, and at the position of the annotation, I clone a sphere so that the gpu can pick it up. When I add the first one, I can see that there is a mesh in the array of detection results. This mesh is the sphere I cloned. After this, I continue to add annotations, but the pick result returns an array containing null. Am I overlooking something? I tried to reproduce it in PG, but it works fine in PG

here is source code from project:
when click i will do

    const annotation = new Sprite(`sprite-${this.currentIndex}`, spriteManager!);
    annotation.position.copyFrom(pos);
    annotation.cellIndex = this.currentIndex;
    annotation.isPickable = false;
    const validateSkinning = (pickInfo.pickedMesh as Mesh).validateSkinning();
    const transformBox = new TransformNode('transformBox');
    transformBox.position.copyFrom(pos);
    const detectMesh = baseSphere.clone(`detectMesh-${this.currentIndex}`);
    detectedList.push(detectMesh);

 private async handleDetected() {
    const { picker, detectedList, scene } = this;
    const canvas = RenderMgr.shared.canvas!;

    const rect = canvas.getBoundingClientRect();

    const width = rect.width;

    const height = rect.height;

    const viewPort = this.camera.viewport.toGlobal(width, height);

    const transformMatrix = scene.getTransformMatrix();

    const xy: IVector2Like[] = [];

    detectedList.forEach((item) => {
      const screenPosition = Vector3.Project(item.getAbsolutePosition(), Matrix.Identity(), transformMatrix, viewPort);

      const x = screenPosition.x >> 0;
      const y = screenPosition.y >> 0;

      xy.push({ x, y });
    });
    const result = await picker.multiPickAsync(xy);
    console.log('result', result?.meshes);
  }

Here is PG:https://playground.babylonjs.com/#W2D1C2#22

image
image1920×1003 151 KB

image
image1920×1017 153 KB

2

I found the problem. When dpr is not 1, multipickAsyc may produce unexpected results. I tried several ways to fix it, but all failed.

Here is how I tried(all failed):

  1. use 1/hardwareScalingLevel and multiply it by the calculated screen coordinates
        let dpr = 1 / engine._hardwareScalingLevel
        async function handlePicking() {
            const transformMatrix = scene.getTransformMatrix();
            const viewport = camera.viewport.toGlobal(engine.getRenderWidth(), engine.getRenderHeight());
            const xy = []
            detectedList.forEach((item) => {
                const screenPosition = BABYLON.Vector3.Project(item.getAbsolutePosition(), BABYLON.Matrix.Identity(), transformMatrix, viewport);
                const x = (dpr * screenPosition.x) >> 0;
                const y = (dpr * screenPosition.y) >> 0;
                xy.push({ x, y })
            })

            const reuslt = await gpuPicker.multiPickAsync(xy)
            console.log('result', reuslt?.meshes)
        }
  1. use canvas.getBoundingClientRect() to get witdh and height pass in the function that calculates the screen coordinates as the viewport
  2. do nothing and use engine.getRenderWidth() and engine.getRenderHeight() directly

This is the PG where the problem reappears：https://playground.babylonjs.com/#W2D1C2#25

3

cc @roland who masterfully created the multipick part :slight_smile:

1 Like
4

see https://forum.babylonjs.com/t/gpu-picker-multi-pick-adjust-to-dpr/58091/3

1 Like
5

Forgive my silence, @Deltakosh my Lord!

The Dark Jedi’s voice crackled through the comm, edged with static and smoldering defiance wrapped in respect. "The call came, and I heard it—but the shadows held me fast. I was entangled in… revelations. New paths. New power. Not just for your dominion, but for all who walk the edge of light and darkness.

I f*cking love ChatGPT! LOL

Simply put, I’m working in parallel on my daily project and on the new RecastNavigationJSPlugin. I’ve added teleport support, but right now I’m struggling to get it working with an imported .bin navmesh. In addition, I’m preparing examples for the new plugin.

Although it’s challenging now, working on the new plugin will be well worth the effort in the long run.

1 Like