Issue picking sprites with OnPointerOverTrigger and a custom pointerMovePredicate

[tibotiber]

Hello,

I read the following thread with great interest a while back:

Mixing onPointerObservable with ActionManager not working

I dug into this quite a bit more and learned some things from the honorable @Deltakosh. @bghgary is completely right about this. The sprite’s don’t exist in the depth buffer, so that means the pick operation doesn’t understand z-order. Priority is always given to a mesh when multiple things are picked, so the sprite is never actually “picked” in your original method…so the executeCode action never fires. The pick keeps thinking that a mesh is being selected, not the sprite. So you then need t…

I was wondering then why the problem of sprites not being picked does not existing with OnPointerOverTrigger which always fires?

Today, I’ve customized my scene’s pointerMovePredicate method to avoid picking meshes through grounds (they’re clearly identified in my code). It works great, but the moment I add grounds to the pointerMovePredicate, it fails to pick sprites if there is a ground behind it.

The pointerMovePredicate is inspired by this post:

OnPointerOverTrigger triggers when mesh hidden by others

I don’t know if it is a bug or a wanted behaviour. But the OnPointerOverTrigger is triggered even when the mesh is not visible because it is hidden by another mesh. But I have the problem only with OnPointerOverTrigger and OnPointerOutTrigger because, with OnPickTrigger, the event is not triggered if we can’t see the mesh. So it would be weird and not obvious to have different behaviours depending on the pick event. I found a former topic talking about the same thing: Check Visibility Pointer…

I’d like to understand what’s happening and if there is a way to get around this?

I’m trying to prep a PG for repro, but posting this in case someone already knows what’s hapenning.

Thanks a lot.

[tibotiber]

And here is the PG example:

Babylon.js Playground

This demonstrates that sprites are not “hovered” when they are in front of a mesh that’s part of the pointerMovePredicate, while other meshes are. If you comment out the pointerMovePredicate function, sprites behave as expected.

[PolygonalSun]

Just taking a look at the provided PG, it does look like the mesh is taking priority over the sprite when it comes to move picking. I’m not that familiar with how sprites are handled so let me dig in a little bit and see what’s going on.

[PolygonalSun]

Okay, so I think I know what’s going on. Specifically with Sprites, when the pick is done, it’s also checked if a mesh was picked. If a mesh was picked, the sprite will be explicitly ignored (spriteSceneComponent.ts, Line 340). As far as I can tell, there is no setting that can be changed to modify this behavior. You could, however, modify your pointerMovePredicate to account for this.

// This version should check for a sprite first and return false if there's a sprite in the way.
scene.pointerMovePredicate = function(mesh) {
    // Since picking is done with the untranslated pointer position, let's use that to pick the sprite manually
    const x = scene.unTranslatedPointer.x;
    const y = scene.unTranslatedPointer.y;
    const result = scene.pickSprite(x,y);
    // We'll then add the check if we got a sprite hit.  If we did, we ignore the mesh.
    return mesh.isPickable && mesh.isVisible && mesh.isReady() && mesh.isEnabled() && mesh.name.startsWith('box') && !result.hit;
}

This should check for any sprites first and pick meshes if there aren’t any in the way. You may want to modify this to suit your needs but this might be a good workaround.

Edit: After reading this back, this seems a bit costly to perf so it might be worth it to either perform the sprite pick outside of the predicate or cache the result so that you only pick once per move event.

You could probably accomplish something like this with:

var foundSprite = false;
...
scene.onPrePointerObservable.add((eventData) => {
    const x = scene.unTranslatedPointer.x;
    const y = scene.unTranslatedPointer.y;
    foundSprite = scene.pickSprite(x, y).hit;
});
...
scene.pointerMovePredicate = function(mesh) {
    // We'll then add the check if we got a sprite hit.  If we did, we ignore the mesh.
    return mesh.isPickable && mesh.isVisible && mesh.isReady() && mesh.isEnabled() && mesh.name.startsWith('box') && !foundSprite;
}

[tibotiber]

Thanks a bunch @PolygonalSun, your approach works:

Babylon.js Playground

And this highlights one thing I wasn’t paying attention to: pointerMovePredicate is called for each mesh each time the pointer moves . Thanks for making me realize this.