Suggestions for mapping points to skeleton?

vladmandic · February 25, 2022, 1:24pm

Im using BoneLookController to map laoded skeleton to a set of provided points (Vector3) and it works fine once I figured out nuances of boneLookController

This is more of a general ask for advice, not an issue with BabylonJS

To correctly map input points to skeleton bone look controller targets, input points need to be scaled (x/y/z) - but since pose can be anything (e.g., point coordinate range is not the same if person is standing up with arms streched vs sitting down with arms next to body), there is no clear reference how to scale them to match skeleton

For any given input, I can set x/y/z scale after manually inspecting scene, but no good idea on how to do that automatically - any ideas?

sebavan · February 25, 2022, 7:41pm

@bghgary do you have any idea ?

bghgary · February 25, 2022, 7:56pm

No, I don’t know much about BoneLookController. Maybe a PG illustrating the scenario would help?

vladmandic · February 25, 2022, 10:27pm

Its really too much for PG as it needs all keypoints and fully rigged skeleton to illustrate.
I can post some screenshots here for illustration?

bghgary · February 26, 2022, 12:14am

Can you load a babylon file or a gltf file with the skeleton in it?

It probably won’t hurt.

vladmandic · February 26, 2022, 12:31am

loading skeleton is not an issue, but to demonstrate the problem i’d need to fully rig it with all bonecontrollers and load keypoints data - and do that for couple of different cases to illustrate how scaling impacts different poses.

i’ll do some screenshots tomorrow…

adam · February 26, 2022, 10:34pm

I’m not sure if this is what you are looking for, but have you tried using bone.getLocalFromAbsolute:

github.com

BabylonJS/Babylon.js/blob/master/src/Bones/bone.ts#L1133

      
        
            

                Vector3.TransformCoordinatesToRef(position, tmat, result);

            }

            

            /**

             * Get the local position of a point that is in world space

             * @param position The world position

             * @param tNode The TransformNode that this bone is attached to

             * @returns The local position

             */

            public getLocalPositionFromAbsolute(position: Vector3, tNode: Nullable<TransformNode> = null): Vector3 {

                var result = Vector3.Zero();

            

                this.getLocalPositionFromAbsoluteToRef(position, tNode, result);

            

                return result;

            }

            

            /**

             * Get the local position of a point that is in world space and copy it to the result param

             * @param position The world position

github.com

BabylonJS/Babylon.js/blob/master/src/Bones/bone.ts#L1147

      
        
            

                return result;

            }

            

            /**

             * Get the local position of a point that is in world space and copy it to the result param

             * @param position The world position

             * @param tNode The TransformNode that this bone is attached to

             * @param result The vector3 that the local position should be copied to

             */

            public getLocalPositionFromAbsoluteToRef(position: Vector3, tNode: Nullable<TransformNode> = null, result: Vector3): void {

                var wm: Nullable<Matrix> = null;

            

                //tNode.getWorldMatrix() needs to be called before skeleton.computeAbsoluteTransforms()

                if (tNode) {

                    wm = tNode.getWorldMatrix();

                }

            

                this._skeleton.computeAbsoluteTransforms();

            

                var tmat = Bone._tmpMats[0];

or bone.getAbsoluteFromLocal:

github.com

BabylonJS/Babylon.js/blob/master/src/Bones/bone.ts#L1091

      
        
                    mat.getRotationMatrixToRef(result);

                }

            }

            

            /**

             * Get the world position of a point that is in the local space of the bone

             * @param position The local position

             * @param tNode The TransformNode that this bone is attached to

             * @returns The world position

             */

            public getAbsolutePositionFromLocal(position: Vector3, tNode: Nullable<TransformNode> = null): Vector3 {

                var result = Vector3.Zero();

            

                this.getAbsolutePositionFromLocalToRef(position, tNode, result);

            

                return result;

            }

            

            /**

             * Get the world position of a point that is in the local space of the bone and copy it to the result param

             * @param position The local position

github.com

BabylonJS/Babylon.js/blob/master/src/Bones/bone.ts#L1105

      
        
            

                return result;

            }

            

            /**

             * Get the world position of a point that is in the local space of the bone and copy it to the result param

             * @param position The local position

             * @param tNode The TransformNode that this bone is attached to

             * @param result The vector3 that the world position should be copied to

             */

            public getAbsolutePositionFromLocalToRef(position: Vector3, tNode: Nullable<TransformNode> = null, result: Vector3): void {

                var wm: Nullable<Matrix> = null;

            

                //tNode.getWorldMatrix() needs to be called before skeleton.computeAbsoluteTransforms()

                if (tNode) {

                    wm = tNode.getWorldMatrix();

                }

            

                this._skeleton.computeAbsoluteTransforms();

            

                var tmat = Bone._tmpMats[0];

vladmandic · February 26, 2022, 11:07pm

@adam i don’t think that would help

my core issue is in order for bonelookcontroller to work properly, target has to be set to a point after bone-end (it should definitely be on the same vector).

but if my target points are not scaled properly, resulting skeleton pose will be incorrect.
for example, hand may be twisted inwards instead of outwards since target keypoint for palm is too close to wrist.

sounds trivial? the problem is that for different sets of input keypoints there seems to be a totally different ideal scale that should be applied. sometimes its scale.x by 2x, sometimes its scale.y by 2x, etc.

and i have no idea what’s ideal scale until i actually do the mapping and see how it looks on the screen. question is for ideas on what to base scale decision on?

adam · February 26, 2022, 11:56pm

I guess I dont understand the issue.

I thought you wanted to transform points from world space to local space or vice versa.

A playground would be helpful.

vladmandic · March 2, 2022, 2:47pm

consider the question as closed (as there is no option to mark it as such).

i can’t do a full playground illustration and without it, it doesn’t make sense to continue - thanks for trying!

fyi, i’ve implemented a “close-enough” algorithm that scales vectors of bone lookcontroller targets by comparing two prominently visible points (e.g. left and right knee) and comparing
a) their bone position vector difference to
b) anchor points vector difference.

and using the ratio of the two to re-scale bone lookcontroller targets

Topic		Replies	Views
How to make arm point at an object? Questions	7	842	July 27, 2020
BABYLON.BoneIKController With GLTF Mesh Questions	52	1881	March 14, 2022
BoneLookController and Bone orientation Questions	19	1031	February 25, 2022
How to recalulate skeleton? Questions skeleton	2	376	March 3, 2023
Skeletal Animation + Babylon Animation Questions	5	632	October 12, 2020

Suggestions for mapping points to skeleton?

Related topics