USD 3000 reward - Anyone figured out retargeting yet?

If you have an existing animation you can clone the animation and set the target to bones of another skeleton. This is essentially how retargeting is done in BabylonJS. However, this applies relative rotations not absolute rotations. This means if you are retargeting to a skeleton that has bones with different orientation and roll, they will rotate differently to what you expect.
I can demonstrate with this video where I show retargeting working as expected in Blender, then I show how it actually works with BabylonJS.

Another solution is to change the role/orientation of the bones to match the animation skeleton but as you can see in this old thread How to change roll of bone? - #21 by ozRocker no one can figure out how to do it.

So I’m wondering if anyone has manged to retarget like how it does in Blender.

Hello! @PatrickRyan might have an idea but he’s on well deserved vacations now so he won’t be able to see this right now :christmas_tree:

Thanks a lot for starting back the thread @ozRocker. I have seen the tweets and we started discussing this back with @bghgary on Friday.

It seems like you are not looking for a simple retargeting as you mentionned.

How is this supported in unity/unreal and such ? I can not remember the setup for it ?

@bghgary any idea how and if we could support it ?

1 Like

I don’t know much about unity/unreal, but you can see in the video how Blender does it. Looks like its using absolute rotation but in BabylonJS its relative rotation

1 Like

This is a good writeup from the godot engine. I believe ue4, unity, and three have similar models. three states in their docs their redesign is based on ue4 and unity. also, some, make some generalizations assuming its a humanoid model. I think blender rigify or one addon has pre-defined skeleton shapes for not only humans but also creatures. I think that could be an interesting expansion

Let s check with both @PatrickRyan and @bghgary after the vacation break what is the best approach as I am fully animation n00b on my side :slight_smile:

Hey Sebavan, thanks for your help. Is there a good day that I can follow up here so I can mark a reminder?

I guess give them a week once they are back to unstack mails and forum so lets say about mid Jan and do not hesitate to ping us back here.

1 Like

Heya, any updates on this?

wont be before mid Jan as stated previously as ppl are still in vacation for a bit.

Does anyone know how rotation info is stored in the GLB file? Is it relative rotation or absolute rotation? I have a feeling its stored as relative which is why the animation keyframes store relative rotation values

All node transforms are local transforms, i.e., relative to the parent node. The animation values modify these values and thus are also relative.

https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html#transformations

ah right. So I guess we can get global rotation by adding the local rotation to the bone orientation. Then the animation keyframes for rotation would now be global and applying that to a different skeleton would always yield the same result. But then the animation playback function would have to change to apply global rotations to the bones. This sounds simple in theory but maybe not possible to implement?

I’ve created a playground to demonstrate the problem. The worm on the left is the base and its animation is being copied to the worm on the right. The initial Y rotation of the target worm bone is 90 degrees different to base worm so after retargeting the target worm gets twisted. Can this problem be fixed if we use global rotation?

Hope this can help !!!

@bghgary and @PatrickRyan let s check how we can support such kind of features

2 Likes

Isnt there issues with global transforms in large worlds though? I think those issues are especially relevant for the use case of avatars in an open world scenario, which seems to be one of the primary use cases.

Not sure what you’re meaning here, mind expanding?

I’m still struggling to get this to work…

So suppose you have an avatar in a large world at some xyz coordinate. If the transform is applied to these potentially numerically large global coordinates, as opposed to numerically small relative coordinates, it can cause issues. I vaguely remember an issue on github where three changed from relative to global some time ago and broke people.

You can look into Floating Origin Template for BJS 5.x - Demos and projects - Babylon.js (babylonjs.com) for the large coordinates thing?