Issue loading/retargeting animations

[peterdijkstra]

Hi,

For a client I need to be able to load any Ready Player Me model and apply an animation to it. Loading the RPM model is easy since it’s just a GLB file, but I am having trouble playing/retargeting animations on them. I am using Mixamo animations, which use the same bone structure, and should work according to RPM. I imported the Mixamo animation in Blender and exported als glb.

The most straightforward way seems to SceneLoader.ImportAnimations(Async). The animation works, but the bones are all stretched out.

I’ve tried all kinds of things, but I can’t seem to get the bones right. I assume something could be done with the targetConverter parameter, but I can’t find documentation on how that is supposed the work.

I’ve also tried just brute forcing replacing targetedAnimation targets, but I can’t get that to work either. (from here)

For what it’s worth, the Mixamo animation plays fine on it’s own mesh:

When inspecting the target of the first targetedAnimation entry in both the imported animation, and the loaded Mixamo mesh + animation, I get wildly different _absolutePositions.

SceneLoader.ImportAnimations:
_x: 24.706941604614258
_y: 88.59120178222656
_z: 27.140859603881836

SceneLoader.LoadAssetContainer:
_x: 0.7434375882148743
_y: -0.19256991147994995
_z: 1.5141488313674927

The rest of the properties seem to be the same. I tried modifying the scale, but that didn’t do anything. Modifying scaleDeterminant only made things worse, haha.

Any help would be greatly appreciated!

Thanks,

Peter

[sebavan]

Adding @bghgary who is amazing at animation Could you provide a repro in the playground ?

[peterdijkstra]

Thannks for your reply @sebavan!

Babylon.js Playground

Here’s a playground with the big stretchy man!

There’s also a commented out bit for loading the mixamo animation with +y up set in the Blender GLB export. Using that one for ImportAnimation causes the RPM model to be oriented on its side

edit: managed to get the same result in a bit more manual manner: https://playground.babylonjs.com/#JQLEGG#3

I took the retargeting method from the thread I mentioned in my opening post

[bghgary]

I will take a look, but this will likely take a while. Skinning issues always take a while.

[bghgary]

Of course, since I said it’s going to take a while, it doesn’t take a while, haha.

The problem here is that while these two have the same bone structure, the bone matrices are not the same.

For example, changing the head to 1 unit in the X direction for hiphop does this:

While doing the same thing for rpm does this:

Hence these are not compatible.

[bghgary]

Using glTF Tools for vscode, you can see there are big differences in the inverse bind matrices for the glTFs.

These are not in order, but you can see there is at least a scale difference.

hiphop_yup:

/accessors/4
[100.00001, 0.00000, 0.00000, 0.00000], [0.00000, 100.00002, -0.00002, 0.00000], [0.00000, 0.00002, 100.00001, 0.00000], [0.00000, -103.99149, -2.07608, 1.00000]
  [100.00001, 0.00000, 0.00000, 0.00000]
  [0.00000, 100.00002, -0.00002, 0.00000]
  [0.00000, 0.00002, 100.00001, 0.00000]
  [0.00000, -103.99149, -2.07608, 1.00000]
[100.00001, 0.00000, 0.00000, 0.00000], [0.00000, 100.00002, -0.00002, 0.00000], [0.00000, 0.00002, 100.00001, 0.00000], [8.20780, -97.23982, -0.47652, 1.00000]
  [100.00001, 0.00000, 0.00000, 0.00000]
  [0.00000, 100.00002, -0.00002, 0.00000]
  [0.00000, 0.00002, 100.00001, 0.00000]
  [8.20780, -97.23982, -0.47652, 1.00000]
[100.00001, 0.00000, 0.00000, 0.00000], [0.00000, 100.00002, -0.00002, 0.00000], [0.00000, 0.00002, 100.00001, 0.00000], [8.20780, -52.86929, -0.76267, 1.00000]
  [100.00001, 0.00000, 0.00000, 0.00000]
  [0.00000, 100.00002, -0.00002, 0.00000]
  [0.00000, 0.00002, 100.00001, 0.00000]
  [8.20780, -52.86929, -0.76267, 1.00000]

rpm:

/accessors/132
[1.00000, -0.00000, -0.00005, 0.00000], [-0.00000, 0.99947, -0.03246, 0.00000], [0.00005, 0.03246, 0.99947, 0.00000], [0.00000, -1.01898, 0.02305, 1.00000]
  [1.00000, -0.00000, -0.00005, 0.00000]
  [-0.00000, 0.99947, -0.03246, 0.00000]
  [0.00005, 0.03246, 0.99947, 0.00000]
  [0.00000, -1.01898, 0.02305, 1.00000]
[1.00000, -0.00000, 0.00000, 0.00000], [0.00000, 0.99449, 0.10484, 0.00000], [-0.00000, -0.10484, 0.99449, 0.00000], [-0.00000, -1.11007, -0.13033, 1.00000]
  [1.00000, -0.00000, 0.00000, 0.00000]
  [0.00000, 0.99449, 0.10484, 0.00000]
  [-0.00000, -0.10484, 0.99449, 0.00000]
  [-0.00000, -1.11007, -0.13033, 1.00000]
[1.00000, -0.00000, 0.00000, 0.00000], [0.00000, 0.98569, 0.16858, 0.00000], [-0.00000, -0.16858, 0.98569, 0.00000], [-0.00000, -1.22961, -0.20984, 1.00000]
  [1.00000, -0.00000, 0.00000, 0.00000]
  [0.00000, 0.98569, 0.16858, 0.00000]
  [-0.00000, -0.16858, 0.98569, 0.00000]
  [-0.00000, -1.22961, -0.20984, 1.00000]

[peterdijkstra]

@bghgary Thank you so much!! This’ll definitely help me find a fix. I was hoping I could use the Mixamo animations without any modifications, but I guess I should retarget the animation in Blender and then export it. With some luck it’ll work on different RPM models too then - since that’s only thing I cannot change.

Again, thank you very much. I know next to nothing about the inner workings of skinning and now I know a tiny bit about it

[peterdijkstra]

Retargeting the animation in Blender and then exporting it seems to work! The arms are messed up because I couldn’t be bothered yet to get the rest pose right in Blender

[sebavan]

@bghgary you are now my go to person for animation for life Thanks

[peterdijkstra]

Alright, I’ve found the REAL fix

Turns out that in the Ready Player Me Unity SDK, there are two FBX files, male and female animation targets, that you can upload to Mixamo to handle the retargeting there. Then you can download the retargeted animations in FBX format. Export those to GLTF from Blender and they just work!

[Shivam_Malhotra]

@peterdijkstra I have been working on a similar setup wherein I need to use mixamo animations on RPM GLB models. Could you please point out which FBX files are you referring to?
Is it RPM_Avatar_Reference.fbx?

[peterdijkstra]

@Shivam_Malhotra They’re called MaleAnimationTargetV2.fbx and FemaleAnimationTargetV2.fbx, in the Unity SDK download! See Loading Mixamo Animations in Unity - Ready Player Me