Most basic skeleton example

I’m trying to work out how to create the simplest example of a working skeleton that is not imported from a pre-created model. (aka Created purely in code js/ts). No fancy humanoid! I just want two bones that are bound to a two boxes or a single cylinder mesh to form a finger or worm.

Here is what I have so far: https://playground.babylonjs.com/#IF31BI#4

 let s = new BABYLON.Skeleton("skeletontest", "skeletonID", scene);
 let bone1 = new BABYLON.Bone("bone1", s, null)
 let bone2 = new BABYLON.Bone("bone2", s, bone1)

With just this, the skeleton is in the scene debugger, in the skeletons drop down, but I’m unable to get the skeleton debug “lines” to visually appear, so something is missing. Help! Is this because the bones need to be bound to a mesh first in order to show up in debug? Or perhaps, the bone needs position and rotation, but I can’t find examples of how to set this. The bone constructor takes a lot of different types of matrices, but I can’t find examples of how to set them. Please educate me on what the Bone constructor’s API means (what’s the difference between localMatrix, restPost matrix and baseMatrix?).

So first step, try to get my skeleton to show up in debugger. 2nd step, try to learn to layout my bones in 3d space using the proper matrices. Step 3, try to bind the bones to the vertices of the mesh I want to deform.

I can’t even get step 1 to work. Any advice?

cc @bghgary but I also recommend checking out some resources on how bone animation works overrall: LearnOpenGL - Skeletal Animation, Tutorial 38 - Skeletal Animation With Assimp (ogldev.org), OpenGL Skeletal Animation Tutorial #1 - YouTube

Doing this by hand is difficult, especially with so many vertices for the cylinder.

Here is the most basic one I can come up with: https://playground.babylonjs.com/#IF31BI#9

You are missing a few things.

  1. The mesh.skeleton needs to be assigned to the skeleton.
  2. The localMatrix parameter of the bone is not actually optional. You must pass at least this parameter or the baseMatrix parameter for the bone to compute its matrices. We should probably update the documentation for the constructor.
  3. There must be bone indices (called MatricesIndices in Babylon.js) and bone weights (called MatricesWeights in Babylon.js) attributes for each vertex in the skinned mesh.

This just makes everything identity, so the bones don’t actually do anything, but this is where you can start.

I would suggest you start with a simpler mesh geometry to understand what is happening. Learning from the skinning section in glTF tutorials may help.

2 Likes

Doing this by hand is difficult, especially with so many vertices for the cylinder.

How about just a box then? Can I assign the north face to one bone and the south face to another bone?

Thanks for the playground, how can I get the skeleton to show up in the bone debugger?

You must bind bones to vertices, not faces. “Bones” are really just an abstraction for matrices; they describe a transformation that should be applied to the vertex. Once a skeleton is bound to the mesh (it has been “skinned”) the vertices will be transformed in the same way that the bones they’ve been mapped to are.

If affected by multiple bones, the bone weights will be used to compute a single transformation matrix for the vertex, factoring in each bone with its respective weight. Animations may however override these (IIRC).

For a better explanation, see this video: How It Works - Skeletons!


For what it’s worth, I agree that the Bone constructor’s arguments are very confusing and have in fact stumbled over them myself, as well as seen others face the same issues. Unfortunately, no one has been able to clear up what exactly all those parameters are doing and I didn’t really understand it from the source code, so I’ve been unable to add the relevant information to the documentation myself.

3 Likes

@bghgary @PirateJC I think we have an opportunity here to improve on the Bones documentation :smiley:

2 Likes

Yup sounds like it. Added an issue for it:

2 Likes

excuse me, is there an update to this demo.

Sorry, there aren’t any new examples yet.

Is there any other reference for the data format reference (matricesWeights, floatIndices) that can implement bone binding to skin in Babylon?

By referring to threejs, I implemented a basic bone animation based on cylinder geometry, but I can’t view the bound bones through the bone viewer (basic skeleton demo )
----------------- split line ----------------
var createScene = function () {

var scene = new BABYLON.Scene(engine);
var light = new BABYLON.PointLight("Omni", new BABYLON.Vector3(10, 0, 10), scene);

var camera = new BABYLON.ArcRotateCamera("Camera", Math.PI / 2, Math.PI / 3, 180, BABYLON.Vector3.Zero(), scene);
camera.attachControl(canvas, false);
const segmentHeight = 30;
const segmentCount = 3;
const height = segmentHeight * segmentCount;
const halfHeight = height * 0.5;

const sizing = {
    segmentHeight,
    segmentCount,
    height,
    halfHeight
}

const cube = BABYLON.MeshBuilder.CreateCylinder("body", {
    height,
    diameterTop: 10,
    diameterBottom: 6,
    subdivisions: segmentCount
}, scene);
const material = new BABYLON.StandardMaterial('cube-mtl');
cube.material = material;
material.wireframe = true;

const vertexTotal = cube.getTotalVertices();
const vertixData = cube.getVerticesData(BABYLON.VertexBuffer.PositionKind);
const indicesMatrix = [];
const weightMatrix = [];

let vertex = BABYLON.Vector3.Zero();

for (let i = 0; i < vertexTotal; i++) {
    BABYLON.Vector3.FromArrayToRef(vertixData, 3 * i, vertex);
    const y = (vertex.y + sizing.halfHeight);
    const skinIndex = Math.floor(y / sizing.segmentHeight);
    const skinWeight = (y % sizing.segmentHeight) / sizing.segmentHeight;
    indicesMatrix.push(skinIndex, skinIndex + 1, 0, 0);
    weightMatrix.push(1 - skinWeight, skinWeight, 0, 0);
}
// console.log(indicesMatrix,weightMatrix)
cube.setVerticesData(BABYLON.VertexBuffer.MatricesIndicesKind, new Float32Array(indicesMatrix));
cube.setVerticesData(BABYLON.VertexBuffer.MatricesWeightsKind, new Float32Array(weightMatrix));
let skeleton = new BABYLON.Skeleton("skeletontest", "skeletonID", scene);
let bone1 = new BABYLON.Bone("bone1", skeleton, null, BABYLON.Matrix.Identity())
let bone2 = new BABYLON.Bone("bone2", skeleton, bone1, BABYLON.Matrix.Identity())
let bone3 = new BABYLON.Bone("bone3", skeleton, bone2, BABYLON.Matrix.Identity())
let bone4 = new BABYLON.Bone("bone4", skeleton, bone3, BABYLON.Matrix.Identity())
bone2.position.y = segmentHeight;
bone3.position.y = segmentHeight;
bone4.position.y = segmentHeight;



cube.skeleton = skeleton;
const viewer = new BABYLON.Debug.SkeletonViewer(cube.skeleton, cube, scene, false, 3, {
    displayMode: BABYLON.Debug.SkeletonViewer.DISPLAY_SPHERE_AND_SPURS
});
viewer.isEnabled = true;
scene.registerBeforeRender(() => {
    skeleton.bones[0].rotate(BABYLON.Axis.Z, .001);
    skeleton.bones[1].rotate(BABYLON.Axis.Z, .001);
    skeleton.bones[2].rotate(BABYLON.Axis.Z, .001);
    skeleton.bones[3].rotate(BABYLON.Axis.Z, .001);
})
return scene;

}

Navigating through the skeleton viewer, I see this bone direction (which is the subtraction of a bone anchor point with its child) is a zero vector in your example: Babylon.js/skeletonViewer.ts at master · BabylonJS/Babylon.js (github.com). I’m not very experienced with bone animation math, so I’ll loop @bghgary in to see if that’s something that makes sense or not.

Unfortunately, due to performance reasons, bone positions cannot be set the way you are doing it. The bone position has to be set as a property (see update PG below). I also changed it to show lines since I think spheres and spurs won’t show correctly with an identity bone matrix.

PG: https://playground.babylonjs.com/#D4ZZ8#358

1 Like

I’ve been studying the awesome playgrounds above and also deep diving into what the babylon file loader does when parsing a serialized scene. My thinking was that if I could understand how babylon was creating the mesh and skeleton from a string, then maybe I could understand how to do it with the babylon skeleton apis.

But I’m still stuck trying to understand the vertex weighting and especially the vertex matrix indexing.

First, I tried to be very precise when creating the blender file. A default cube is 2M in all dimensions, I scaled it 6M high, applied all transformations, did a loop cut 5 times to add segments, created an armature (bone), (by default it “faces” up, so I left it that way). divided that 5 times and placed it perfectly inside the cubes by setting lengths to 2 then bound mesh and armature with automatic weighting. I exported this blender scene to .babylon file using the plugin and in the playground below I pasted the raw data into the source so that we can inspect what babylon is parsing.

Next I tried to reproduce an exact mesh using the babylon mesh builder. The createBox MeshBuilder does not have an option to segment the box, but it turns out you can use a cylinder (which has an option to create segments) and reduce the tessellation to 4, which will create a box like column.

In the following playground you will see two “boxes” side by side. One is created by parsing the serialized data that comes from blender. The 2nd one is created manually using babylon mesh builder.
https://playground.babylonjs.com/#HFWCMJ#1

It turns out creating the skeleton and making it appear in the mesh isn’t the hard part. Indeed it turns out if you use an identity matrix on the 2nd bone on… then the Sphere and Spurs display does not appear. But if you pass it a translated matrix the sphere and spur display appears (and that’s what the blender export is providing the Skeleton Parse method). But it also seems like the skeleton is only responsible for some hierarchy information. It doesn’t describe how it is influencing the mesh. That heavy lifting seems to be on the mesh itself.

The .babylon serialization for a mesh contains two keys probably related to this: matricesWeights and matricesIndices.

The length of the matricesIndices array matches the total vertex count (110) of the imported mesh, so my intuition tells me that the matricesIndices should specify something regarding each vertex in the mesh, but I can’t make heads or tails of what the value means. The values are all integers, and out of the 110 length array, there are only 7 unique values used:

[...new Set(data.meshes[0].matricesIndices)] => [4, 1027, 0, 256, 131328, 197121, 262914]

But why 7? There are 5 bones, so if you wanted to specify that you were under the influence of one of those bones, you could just use index into the skeleton hierarchy. But… that wouldn’t be sufficient to indicate that a vertex is under the influence of multiple bones… for that the value needs to represent a spread of choices… maybe… :person_shrugging: I can’t figure it out. Does anyone know?

matricesWeights is also an interesting array. The length is 4 times the length of the vertex count. After watching the babylon video on skeletons, I’m told that the default weighting of bones is “up to 4” (but can be “pushed to 8”). So I’m guessing that this array is telling me the distribution of weights for each vertex on 4 bones. But which 4 bones? I have 5 bones, so how do these weights correspond to the right bones in the hierachy? That must be what the matricesIndices number is about. Also interesting is that if you look at the array in batches of 4, the last number is always 0, while the first 3 are the only ones used and always add up to nearly 1. This observation seems to correlate with the numBoneInfluences = 3 in the imported data. :person_shrugging:

What do these numbers mean? :pray:

I couldn’t find the source code that actually slurps in the matricesIndices part of the .babylon file. If someone can point me there that would be helpful.

After looking at these mysterious numbers a bit longer like a crazy person, there is something suspicious about them. Like they feel like they have some binary encoding, so I printed out the matricesIndices values along side it’s corresponding matricesWeights and noticed that small values like “0” and “4” only ever had contribution from one position in the weights array.

“0” could mean all my influence comes from bone with index 0. And corresponding weights only come from one of the 4 positions: [0.98, 0, 0, 0]

“4” could mean all my influence comes from bone with index 4. And weight looks like: [0.98, 0, 0, 0]

“256” in binary is 00000001 00000000 which could mean my influence is bones with index 1 and 0, with weight example: [0.91, 0.08, 0, 0]

“1027” in binary is 00000100 00000011, which could mean my influence is bones with index 4 and 3. with weight example: [0.08, 0.91, 0, 0]

131328 in binary is 00000000 00000010 00000001 00000000 … which, ignoring the leading zeros could mean influence from indexes of 2, 1 and 0. Sample weight: [0.09, 0.82, 0.08, 0]

197121 in binary is 00000000 00000011 00000010 00000001, ignoring leading zeros is 3,2,1. Sample weight: [0.08, 0.82, 0.08, 0]

262914 in binary is 00000000 00000100 00000011 00000010, ignoring leading zeros is 4, 3, 2. Sample weight: [0.08, 0.82, 0.09, 0]

hmmmmm… :mag:

Ok I think I figured it out.

Here’s an updated playground with weights that I have manually set:

https://playground.babylonjs.com/#HFWCMJ#2

Through experimentation, the way I think it works is that for each vertex on the mesh you create a sort of mapping like this:

index = [0, 1, 2, 0];
weight = [0.25, 0.7, 0.05, 0];

These mappings need to have a width of length 4, but not every “slot” needs to be meaningful. In the first array you can put the indexes of the skeleton in any positions of that array, again specifying up to 4. Then in the subsequent weights array fill in the influence of each position, to turn the bone influence up or down, corresponding to the value in the first array.

So this:

 index = [0, 0, 0, 0];
 weight = [0, 0, 1, 0];

does the same thing as this

 index = [0, 1, 2, 3];
 weight = [1, 0, 0, 0];

Because they both say, bone at index 0 in the skeleton is turned up to 100%.

Finally flatten all the mappings into 2 arrays and load them into the mesh. Please let me know if I got anything wrong.

Yes, you got it right :slight_smile:

The matrix indices list all the bone indices that influence a given vertex, and the matrix weights give the relative weight of each bone for that vertex. The sum of all weights must equal 1. As you discovered, if a vertex is influenced by less than 4 bones, you must set some weights to 0 to account for this.