Ray picking on meshes deformed by skeletons

Im trying to get a ray picking to work on meshes that are deformed by skeletons.
Is this generally possible? And if yes, how?

I created a test playground here: https://playground.babylonjs.com/#8BVNBC#8
(use arrowkeys to move around. Mouse to look)

  1. The raycollision check seems to work, but for me only with the not transformed cube. If I look at the origin of the scene, I get a console.log that Im hitting the cube. Trying to look at the moving(transformed) box it doesnā€™t give me any feedback.
    Whatā€™s happening? Is the ray collision calculated before the skeletal animation is applied?

  2. I encounter another problem with the visibility of the mesh aswell. When I move the camera to a position where the origin is not visible anymore and look to the moving cube. It disappears.
    Any idea how to keep the the transformed cube visible?

  3. After a while playgrounds fps drops. Tested on chrome, firefox, windows10 and mac OSX.
    Has anyone an idea what Iā€™m doing wrong? I noticed the same behavior with this PG example, here:
    https://www.babylonjs-playground.com/#KNE0O#84

Im fairly new to coding and babylon - coming from a 3D-Animation background. So please expect some stupid mistakes. ^^
Thanks in advance!

As far as I know a skeletal (aka ā€œskinā€) deformation of a mesh occurs on the gpu, meaning that its position is not changing on the cpu which is where our ray cast takes place. Perhaps someone with more knowledge can confirm this or suggest a solution. There is a mesh.computeBonesUsingShaders = false option which might change this, though I have not tried it. The game Iā€™m working on has very simple block characters, and the animations are just rotating the arms/legs directly in javascript-- this much I can confirm produces movement that is detectable with a raycast. Iā€™m not sure how to make animations this way in animation software (nor if importing them would work). I too, would be curious if anyone knows.

Iā€™m not sure about #2. Iā€™ve see it myself. Thereā€™s an option to keep a mesh always visible with respects to the camera, mesh.alwaysSelectAsActiveMesh = true which might be needed if the mesh is traveling via an animation as opposed to having its position actually move. Iā€™m not sure if the real answer here is ā€œdonā€™t move a mesh super far via a bone animation, instead move the actual positionā€ or if there is a setting to change.

As far as #3 goes, the fps drop is because the rays are being endlessly calculated, and then rendered as some sort of 3d object via the helper. Eventually there are many thousands of them in the scene, which slows the rendering. Hereā€™s a small change that cleans up the rays one second after theyā€™re created, which will keep the FPS high: https://www.babylonjs-playground.com/#KNE0O#145

1 Like

Hey Alex, thanks a lot for helping!

I tested your suggestions and restructured the playground(together with a friend) to make it easier to understand.

https://playground.babylonjs.com/#8BVNBC#34

In the viewport from top to bottom:

  1. row: 'skinnedā€™cube, the wireframed cube is just a placeholder for the position of the 'skinnedā€™cube in restposition. - pickWithRay working on restposition but not transformed mesh
  2. row: A placed mesh in blender - pickWithRay working
  3. row: animated cube(no skinbind) - pickWithRay working

mesh.alwaysSelectAsActiveMesh = true; (used in line 38) is working great. Now the mesh doesnā€™t disappear animore. Thanks @timetocode !

mesh.computeBonesUsingShaders = false; (used in line 39) unfortunately doesnā€™t seem to work for me. It even breaks the raypicking functionality( thatā€™s why I commented it out) Or am I using it wrongly?

A workaround could be to place an extra ā€˜hitboxā€™ to every bone via babylon. Which feels very hacky to me, but Iā€™ll make a test.
Is there really no smarter way? Maybe @Deltakosh has an idea?

Hello @flo and welcome!
@timetocode is right: Because bones are GPU accelerated you cannot pick them with a CPU operation (meaning the deformed geometry is not in the CPU memory but in the GPU memory).

The best way to deal with it is to attach hitboxes to your mesh as you mentioned. This could sound hacky but this is a good viable way to do it :slight_smile:

Here is a good example: https://www.babylonjs-playground.com/#11BH6Z#395

1 Like

I implemented the solution into my first playground and commented:
https://playground.babylonjs.com/#8BVNBC#45

Summary:

  • mesh.alwaysSelectAsActiveMesh = true; - solves the visibility issue
  • Working with a separate mesh .attachToBone(bone, affectedTransformNode) on skeleton animations solves the raypick problem.

I have two further question, if someone has time to answer it would be great:

  • the hitbox seems to lag one frame behind(jittering), is there a way of getting rid of the delay? Or wount it be a problem?
  • From the documentation: .attachToBone. I donā€™t understand why it needs a Ref to the ā€˜sourcemeshā€™:

affectedTransformNode: TransformNode

Thanks for the confirmation and the ā€˜.attachedToBoneā€™ hint, @Deltakosh
And again thank you @timetocode for the answer :slight_smile:
(I mark it as solved, with Alexā€™ answer? not really sure if this is how it works)

For

This is because some skeletons can rely on root mesh for their initial transform

1 Like

Hereā€™s a slight code change that wires up the hitbox once at load time, instead of once every frame:
https://playground.babylonjs.com/#8BVNBC#47

I have no idea about it being delayed by a frame. Iā€™m on a 144 hz screen (<7 ms/frame) and I canā€™t see it during most of the movement. I do see some sort of jump as the cube crosses the middle area, though I donā€™t know what that is.

As for whether a collision (like a raycast in a first person shooter) will work if the character movement is late by frame (is it?) this depends on how fast the object is moving. Generally speaking the answer is that the raycast will fully miss when the object is moving so fast that it does not overlap its previous position from one frame to the next. Thatā€™s pretty fast, and I would say is generally beyond the realistic speed of some sort of character animation. If one is using an animation (instead of actual movement in the engine) to move relatively small fast objects like a bullet or a rocket then itā€™ll be an issue.

Hereā€™s something showing how far the hitbox moves per frame https://playground.babylonjs.com/#8BVNBC#48 ā€“ it leaves a tracer of the hitbox positions every frame. Note: only an accurate visual tool if at maxed frame rate (60fps, etc).

1 Like

Ok Iā€™m stupid.
The cube is exported with 30fps, maybe this is the ā€˜jitteringā€™. And the ā€˜hickupā€™ in the middle could be due to two times the same Keyframe on start and endframe of the animation. Itā€™ll work anyway for my purposes :slight_smile:

Thanks for the improved playground and the explanations!
@timetocode & @Deltakosh

1 Like