Anyone figured out retargeting yet?

ozRocker · January 23, 2023, 10:41pm

This is just for rotation though

Benjythebee · January 30, 2023, 5:48pm

@bghgary did you have any thoughts on my BB playground repro?

bghgary · January 31, 2023, 5:47pm

I haven’t gotten time to look at it yet. I will try to this week.

Benjythebee · February 3, 2023, 10:22pm

Heya, my team and I are willing to put a bounty on this, probably 500 USD.
Is there a process for this?

ozRocker · February 4, 2023, 2:17am

The bounty for this has been moved up to 3000 USD. If there is a formal process of attaching a bounty please tell us so we can do this the right way

bghgary · February 6, 2023, 10:49pm

Sorry, I’ve been sick last week. Thanks for the patience.

Benjythebee · February 8, 2023, 7:04pm

Any luck?

aWeirdo · February 8, 2023, 7:22pm

I don’t know the maths, but i suppose one solution could be to get the difference between the bones rotations (origin and target bones) and apply it to the cloned animation’s keyframe values?

concept, (in euler for readability)
keyframe rotation value (x: 1, y: 0, z: 0)

if bone rotation difference is 180 degrees,
result keyframe value would be (x: -1, y: 0, z: 0), etc

Concept is simular to how transformNode.rotateAround() works for nodes and meshes

github.com

BabylonJS/Babylon.js/blob/88e5d2c559d964ce86474f505015c5cc7601c387/packages/dev/core/src/Meshes/transformNode.ts#L926


      
          /**

           * Rotates the mesh around the axis vector for the passed angle (amount) expressed in radians, in world space.

           * Note that the property `rotationQuaternion` is then automatically updated and the property `rotation` is set to (0,0,0) and no longer used.

           * The passed axis is also normalized. .

           * Method is based on http://www.euclideanspace.com/maths/geometry/affine/aroundPoint/index.htm

           * @param point the point to rotate around

           * @param axis the axis to rotate around

           * @param amount the amount to rotate in radians

           * @returns the TransformNode

           */

          public rotateAround(point: Vector3, axis: Vector3, amount: number): TransformNode {

              axis.normalize();

              if (!this.rotationQuaternion) {

                  this.rotationQuaternion = Quaternion.RotationYawPitchRoll(this.rotation.y, this.rotation.x, this.rotation.z);

                  this.rotation.setAll(0);

              }

          

              const tmpVector = TmpVectors.Vector3[0];

              const finalScale = TmpVectors.Vector3[1];

              const finalTranslation = TmpVectors.Vector3[2];

Just a thought, but i wouldn’t know if it’s feasible

ozRocker · February 8, 2023, 10:48pm

I’m hoping to find a solution that doesn’t require creating (cloning) the keyframe values because that produces a significant frame rate drop if you have a lot of avatars on screen.

jelster · February 9, 2023, 1:08am

I feel like there’s a solution for this problem that involves adding together different transform matrixes…

As discussed upthread, a transform to global (world) space prior to retargeting an animation is needed. The two questions as I see it I are where and how to accomplish it without having to clone keyframes.

Things available (not exhaustive)

the world to local transform matrixes for mesh A and mesh B (presumably)
a local space matrix for each bone position
the base “world view” matrix
key frame data laying out the xform of each bone in relative (model) space

Maybe you could:

add to the main animation loop another step that applies a matrix xform, if one is supplied and/or available for that animation instance. The xform would be composed at the time of the animation re-target. If not present, the identity matrix can be used instead (no-op)
a bit hackish but have you tried exporting from blender as OBJ or other BJS-compatible format, then dropping it into the sandbox or otherwise converting from there to a GLTF?
since it doesn’t appear that gltf uses absolute values, could the key frame data be separated and output from blender as a text/csv file with the world space coordinates you’re looking for? You’d load the data in separately from the meshes of course, and process it to combine and project out the positions and rotations in it onto the new mesh

Hope these help!

sebavan · February 13, 2023, 9:57pm

Retargeting as mentioned here is not supported by the engine, we only support retargeting identical skeletons in shapes and hierarchy. This is too restrictive for your scenario and would be an amazing addition to the framework.

Just to let you know we are making a priority of improving our animation system for our next release. Retargeting as discussed here is part of our improved plan. You can follow the progress here Animation Improvements · Issue #13534 · BabylonJS/Babylon.js · GitHub

Benjythebee · February 23, 2023, 7:22am

Is there a way we can help with the progress on this?

Evgeni_Popov · February 23, 2023, 9:05am

I think what would help is to make some simple repros in the Playground of the various features we need to support (even if they don’t work, of course!), like animation retargetting or moving meshes from one skeleton to another (from another thread).

ozRocker · February 23, 2023, 9:19am

This one demonstrates the retargeting issue. The 2nd mesh gets twisted https://playground.babylonjs.com/#Y3G617#4

Evgeni_Popov · February 23, 2023, 11:40am

The problem in this PG is that the inverse bind matrices are not the same for the bones in worm_base.glb than in worm_target.glb, which means (I think) that the rest pose is not defined in the same way in the DCC tool.

ozRocker · February 23, 2023, 12:44pm

The rest poses are the same its just that with the 2nd object one of the bones has a different roll value than the first object. So I’m guessing rotation is happening relative to the bones roll/orientation but I want it to be rotating in world space so for example “point north” will have the same result for both meshes. In Blender and Unity that’s how it works.

Evgeni_Popov · February 26, 2023, 6:38pm

Unfortunately, I can’t help you more, I’m not yet familiar with the animation system…

sebavan · February 27, 2023, 5:00pm

But this is all planed as a priority as soon as we deliver our next release. Thanks a lot for the example, it will help use nailing this for the future release.

Evgeni_Popov · May 3, 2023, 3:27pm

Here’s what I came up with to retarget an animation group from one skeleton to another:

It works in your case, I’m still doing some testing for different use cases.

Topic		Replies	Views
Looking for an example where a mixamorig model has animations applied to it from any other mixamorig animation Questions	6	548	April 29, 2022
Exporting Animation Questions	17	2132	August 24, 2019
Skeletal Animation + Babylon Animation Questions	5	555	October 12, 2020
How to transfer a human pose from Motion Capture to Humanoid character Questions	4	1837	May 12, 2019
Animation group copy not working in Babylon 5 Bugs	5	656	April 12, 2022

Anyone figured out retargeting yet?

Related Topics