BABYLON.BoneIKController With GLTF Mesh

Yo @Deltakosh and @bghgary … I got a question about using the BoneIKController with a model from GLTF.

When you import gltf the transforms are linked to the bone… You don’t really move bones, but instead move the linked transform and that in turn moves the bones…

So how do i use BABYLON.BoneIKController when the second param is asking for the bone to control?

So guys @Deltakosh , @sebavan , @Evgeni_Popov and @bghgary

I made a class called GltfBoneIkController to try and figure out what needs to be done to support GLTF meshes where the bone has a bone._linkedTransformNode.

Here is a playground were i am loading a GLB (Big file 20MB… so please wait for it to load)

https://playground.babylonjs.com/#D0T14K?UnityToolkit

Pease look at lines 449, 456 and 466 … These are the lines that move the bone after all the calculations. Except we need to move the bone._linkedTransformNode instead of the actual bone.

Note: the bone update will move the actual bone fro the linkedTransformNode at a different spot in the codebase

basically how do i the equivalent calls for:

this._bone1.setRotationQuaternion

and

this._bone1.setRotationMatrix

and

this._bone2.setAxisAngle

but for the linked transform node

i outlined like this. Lines 440 thru 470

if (this._bone1) {
             if (this.slerpAmount < 1) {
                 if (!this._slerping) {
                    BABYLON.Quaternion.FromRotationMatrixToRef(this._bone1Mat, this._bone1Quat);
                 }
                 BABYLON.Quaternion.FromRotationMatrixToRef(mat1, _tmpQuat);
                 BABYLON.Quaternion.SlerpToRef(this._bone1Quat, _tmpQuat, this.slerpAmount, this._bone1Quat);
                 angC = this._bone2Ang * (1.0 - this.slerpAmount) + angC * this.slerpAmount;
                 if (this._bone1._linkedTransformNode != null) {
                    // TODO: PORT TO TRANSFORM NODE - this._bone1._linkedTransformNode.setRotationQuaternion(this._bone1Quat, BABYLON.Space.WORLD, this.mesh);
                 } else {
                    this._bone1.setRotationQuaternion(this._bone1Quat, BABYLON.Space.WORLD, this.mesh);
                 }
                 this._slerping = true;
             } else {
                 if (this._bone1._linkedTransformNode != null) {
                    // TODO: PORT TO TRANSFORM NODE - this._bone1._linkedTransformNode.setRotationMatrix(mat1, BABYLON.Space.WORLD, this.mesh);
                 } else {
                    this._bone1.setRotationMatrix(mat1, BABYLON.Space.WORLD, this.mesh);
                 }
                 
                 this._bone1Mat.copyFrom(mat1);
                 this._slerping = false;
             }
         }
         if (this._bone2._linkedTransformNode) {
            // TODO: PORT TO TRANSFORM NODE - this._bone2._linkedTransformNode.setAxisAngle(this._bendAxis, angC, BABYLON.Space.LOCAL);
         } else {
            this._bone2.setAxisAngle(this._bendAxis, angC, BABYLON.Space.LOCAL);
         }
         this._bone2Ang = angC;

Can you guys take a look and apply your 2 cents on how we should tackle this … Please :slight_smile:

The linked transform node is a TransformNode, so you have the position, rotation / rotationQuaternion / scaling properties that you can modify as you wish + all the methods of the TransformNode class (rotate, rotateAround, etc).

I get that… i guess what i am asking is the best way to modify BoneIKController to handle bones with linked transform nodes ?

I guess both the constructor and update method should account to read data from either the bone or the attach transform node in the other case, maybe @adam who built the IK controller has an idea ?

There are so many references to an actual BABYLON.Bone (bone1 and bone2) thru out the code. It would probably be best to keep the constructor the same and always pass a bone. But then in the update check if that bone has a linkedTransformNode.

I am going to fart around with adding those ROTATIONS functions to support the linked transform node and see what that does

Would be great thanks a lot :slight_smile:

I have a lot invested in the “classic” method, and would prefer that it not be messed with. It is complicated enough already.

1 Like

We need @mrlooi to come flex. That man is the glb IK king.

2 Likes

I talked to CJ last night. We did a google meet to go over a few IK issues and what he did for his company and using IK. First off, he is NOT using BoneIKController. They rolled their own custom IK Solver library. But he did go over our current BoneIKController issues with me a little bit.

I am proposing to NOT to create a separate controller but simply ALTER the three lines of code that actually rotate the bone to check if there is a bone._linkedTransformNode and if so rotate that instead of the bone… the bone will update itself from the linkedTransformNode so we DONT break any existing functionality of the current BoneIKController.

So basically we would need to PORT the:
bone.setRotationQuaternion
bone.setRotationMatrix
bone.setAxisAngle

To do its magic rotate a transform node. @mrlooi says it should really be a matter of setting the absoluteRotationQuaternion of the transform node. There is currently no transformNode.setAbsoluteRotation like there is a setAbsolutePosition … so i think we need to start there… adding support for BABYLONJS to have a setAbsoluteRotationQuaternion. If we had that i think thats all we need for this.bone1._linkedTransformNode rotation. The this.bone2.setAxisAngle we would have to PORT to work on a transformNode. That function looks a bit busy because it has to take the parent transform node into account… I dont know that we need to do all that since we are trying to rotate a transform node instead of a bone.

So it REALLY help if i had someone who knows more about the methods of rotations to help PORT those THREE bone rotation functions listed above.

So if ANYONE is willing to help with this. please contact me and lets get it done :slight_smile:

2 Likes

Ping me back in a week if nobody else is volunteering :slight_smile:

@Pryme8 @MackeyK24 thanks for the mentions :rofl:

On a general note I do think we should aim to remove bone transformation support in BJS entirely, and just use Tnodes everywhere. It’s not a good separation of concerns to have two things trying to do the same thing.

The bones are broken in more than a few places and every single dev (not exaggerating) I know who has worked on BJS bones gave up and used Tnodes instead. I personally wasted months of effort trying to port transformation functions to the Bone class… @Pryme8 knows what I’m talkin about :grimacing:

If you look at Three.js, the bone is just a simple wrapper on top of object3D i.e. transformNode. I believe this should be the way!

Is there a reason why we are trying to polyfill all these transformations into the Bone class? Why not just make it a Tnode.

But that’s a whole separate topic!

@MackeyK24 This is what I have for get + setGlobalRotation, hope it helps!


const TMP_Matrix = new Matrix();
TransformNode.prototype.getGlobalRotation = function () {
	if (this.parent instanceof TransformNode) {
		let PR = this.parent.getGlobalRotation();
		return PR.multiplyInPlace(this.getLocalRotation());
	}


	const scale = this.scaling;
	if (scale.x == 1 && scale.y == 1 && scale.z == 1) {
		return this.absoluteRotationQuaternion.clone();
	} else {
		const sx = 1 / scale.x,
			sy = 1 / scale.y,
			sz = 1 / scale.z;
		//@ts-ignore Access a private field
		const m = this.getWorldMatrix()._m;
		Matrix.FromValuesToRef(
			m[0] * sx,
			m[1] * sx,
			m[2] * sx,
			0.0,
			m[4] * sy,
			m[5] * sy,
			m[6] * sy,
			0.0,
			m[8] * sz,
			m[9] * sz,
			m[10] * sz,
			0.0,
			0.0,
			0.0,
			0.0,
			1.0,
			TMP_Matrix
		);
		return TMP_Matrix.getQuaternion();
	}
	// return this.absoluteRotationQuaternion.clone();
};
TransformNode.prototype.setGlobalRotation = function (R) {
	let P = this.getParent();
	if (P instanceof TransformNode) {
		let parent_GR = P.getGlobalRotation();
		let LR = parent_GR.inverse().multiplyInPlace(R);
		this.setLocalRotation(LR);
	} else {
		this.setLocalRotation(R);
	}
};
2 Likes

I’m assumijg the answer to why not just tnodes will be backwards compatibility.

2 Likes

Yo @mrlooi … I dont seem to see a matrix.getQuaternion or a transformNode.getLocalRotation or a transformNode.setLocalRotation or a quaternion.inverse in the BABYLON JS API at all… hmmm

@mrlooi we d be more than happy to try and fix the bones implem where they have issues.

Adding @Deltakosh the master of the skeleton FYI.

They both Node base classes. So why don’t we just extrapolate one more class above that with all the standardized methods and make the bone and transformNode be extensions of that? Then we wont have to maintain two separate classes with redundant methods.

3 Likes

I think I asked about that ages ago, definately old forum. Think @Deltakosh only wanted methods which worked on all sub-classes. Lights & camera are also Node children, and they have kids of there own.

Edit: Maybe re-base Bone off of Transform Node, then you wouldn’t even a linked transform Node?

2 Likes

Wait sooo, yeah what’s stopping us from adding a transform infront of that node extension!? You’re right, that’s prolly the best.

Actually, I was thinking of changing the base class of Bone to TransformNode, not switching the order of TranformNode / Node in class hierarchy.

Think gaming both ways before making such a big change is probably required. Earlier AbstractMesh base class was changed to TransformNode from Node. If that is not for Bone, I think you would have to change AbstractMesh to match.

2 Likes