How to transfer a human pose from Motion Capture to Humanoid character

Questions
1

I am currently doing 2D Pose Estimation task in Deep Learning (Similar to motion capture) and I would love to visualize it on a humanoid using Babylon JS. I am new to Babylon JS but I have previous experience with Unity 3D. There are lot of resources for pose retargeting using the GUI but not much for doing it in code. I did google it but nothing significant came up. I hope someone could help me with how I should go about it.

1 Like
2

ping @ozRocker punkoffice

3

Create a rigged avatar with the same skeleton the Pose Estimation framework is using. Make sure they have the same rest poses. I work from Blender 'cos the exporter from there is good In code you can dynamically apply new animations to your avatar like this:

  1. Load .babylon avatar (mesh and skeleton)
  2. Load .babylon animation file (mesh, skeleton and animations)
  3. Copy animation - avatar.skeleton.copyAnimationRange(animator.skeleton, “Dancing”, true);
  4. Playback animation - avatar.skeleton.beginAnimation(“Dancing”,true,1)

Ideally you’d want the animator file to only have skeleton and animations but the Blender exporter won’t allow exporting without a mesh, so export as usual, then you will have to remove the mesh data inside the .babylon file manually (or you could just edit the Blender exporter if you have those skills)

I clean out the mesh so it looks like this:

“meshes”:[{“name”:“Animator”,“id”:“Animator”,“billboardMode”:0,“position”:[0,0,0],“rotation”:[0,0,0],“scaling”:[1,1,1],“isVisible”:true,“freezeWorldMatrix”:false,“isEnabled”:true,“checkCollisions”:false,“receiveShadows”:false,“skeletonId”:0,“numBoneInfluencers”:6
,“positions”:[]
,“normals”:[]
,“uvs”:[]
,“matricesWeights”:[]
,“matricesIndices”:[]
,“matricesWeightsExtra”:[]
,“matricesIndicesExtra”:[]
,“indices”:[]
,“subMeshes”:[{“materialIndex”:0,“verticesStart”:0,“verticesCount”:0,“indexStart”:0,“indexCount”:0}]
,“instances”:[]}

2 Likes
4

Come to think of it, maybe you can simply remove all vertices from the mesh within Blender instead of doing so within the text file. I haven’t tried it so I’m not sure if it’ll mess with the weighting of the skeleton or not, but its worth a shot.

1 Like
5

that was brilliant.

I’m heading getting back to blender skeleton rig exports soon. Created a “Surfing”.

“Dancing” && “Surfing” istypeof cool

So that was a helpful concept. Thx.

Blender exporter is excellent. Can’t wait to get back in that pipeline.

as for MOTION-CAPTURE… people keep asking about this!

That is three times in my randomness counter.

So using my ONE phone-a-friend …bat-signal. (>8<)

has anyone interfaced KINECT into BABYLON.skeletons (yet)?

I googled it and it looks like there were some projects - maybe that would be a good start.

It would be remarkable. : )