Adding a drag behavior inside of a scene.onPointer down

[JCPalmer]

I am trying to move my IK based, armature posing from Blender directly into BJS. When using Blender, I interactively positioned the IK bones to get the figure as desired. I then stored the pose in a Blender Pose Library. I wrote a Blender procedure to generate a small JS file with the world matrices of all bones, excluding bones at rest & IK bones, for each pose in the library.

I can animate different limbs (sets of bones) independently in my system, including the body. Well almost . Since the matrices are hard coded, they depend on the bone immediately preceding to be in rest position. Makes for at best a very stiff performance. At worst, total cluster fuck.

Think about clapping, then think what would happen if the armature was sitting down when the 2 animations (left hand & right) were run. Hopefully you see the problem, & I am not boring you.

Just storing the rotations instead of the matrix is not a solution. That means you will loose the ability to have independent limb animation, which is an absolute requirement of mine. Think about the animation of turning a light switch on, but now the torso is now turned. You are going to miss unless you store the rotations of every bone. Bye-bye independence.

If the animation were based on the IK target mesh position (& rotation for the wrist) and the position of the pole mesh, then piece of cake, or close except for extremes.

Continuing to use Blender, & exporting the IK data is not going to work. The systems are too different, so I starting an effort in a tool scene.

The red squares are the pole targets for the left & right arms, respectively. The wrist donuts are both the IK target mesh, and function to record the rotation of the wrist, see below.

It is operational, not actually recording anything yet.

Now the problem, trying to control 3 vectors with controls is almost impossible.

Blender allows drag. If you press either the X, Y, or Z keys you can control the direction. Is it going to be possible to add a BABYLON.PointerDragBehavior from inside a scene.onPointerDown? If not, I guess I will have to have controls to indicate which pane or direction, and not a key.

Sorry, for the long question, but seems like others are do stuff in this area right now, so I gave a little more detail.

[JCPalmer]

One of the reasons I wrote such a long question was it was to myself as well. In Blender you can also do sequence of a key down of the action (rotate or grab), then the direction key. The mouse then performs the positioning / rotation of the previously selected object.

I am assuming that you cannot do a drag outside of having the mouse down, but I think this might work: When a X, Y, or Z key is pressed, set a drag behavior on all the control meshes. Who cares if only one gets used. Then things should work well with the correct drag installed in advance.

Given that the drag behavior is going to be changing, I’ll just have add this to the loop of behavior adds or it probably is only going to work once:

controlMesh[i].remove(controlMesh[i].getBehaviorByName('need to know yet'));

[JCPalmer]

While this PointerDragBehavior method on every control mesh works, see Babylon.js Playground, I have been looking at the source for PointerDragBehavior .

It has methods startDrag and releaseDrag. If I can call these, actually triggered by a key press, and ended with mouse click, then I can do these operations outside of a drag.

I’ll do a repo src directory search for those 2 to see what args are set to, but probably also need to override addBehavior called on the mesh subclass I made for control meshes to actually fall short of turning it on. Otherwise, the click to stop drag will probably not work.

Problem is, I cannot find source for that. Searched entire directory in File explorer, and only getting hits in gizmos dir. Anyone know source for addBehavior?

[JCPalmer]

Ok, found addBehavior in node.ts. Changing this is not the way to go. The Attach method of PointerDragBehavior is where the onPointerObservable is added & everything happens.

My next move is to just grab that & put attach directly in my mesh subclass, then change it.

[JCPalmer]

Ok, sometimes you just gotta start ripping stuff out to really figure things out.

First, the options specified in the constructor of PointerDragBehavior can be set later, so all that remove stuff in above posts is not really required, just change the option.

A PointerDragBehavior can be simulated without clicks, using startDrag, using a fictitious pointer ID. This should not have a conflict with terminating with a mouse click, because:

• StartDrag sets _moving = true, so any subsequent mouse down over the mesh will get ignored by the internal pointer observable.

• any mouse up is going to get ignored, since the point ID is not going to match the fictitious one of the manual start.

• mouse moves work as if initiated with a click over.

I consolidated the options set, additional start methods to be called by key presses, {x, y, z}, anda scene level mouse down to terminate the operation, all into a small sub-class of PointerDragBehavior.

My editor will have a concept of a currently selected mesh, using another scene.onPointerDown. If one of the controller meshes is picked,

• Record the mesh as the one & maybe highlight it.
• A x,y, or z key calls the corresponding moveAlong() of the behavior of the selected mesh.

If this works, will have to set the scene.onPointerDown back to the mesh picker, not null, as it currently is.

module QI {
    export class PointerMoveBehavior extends BABYLON.PointerDragBehavior {
        private static _X_AXIS = new BABYLON.Vector3(1, 0, 0);
        private static _Y_AXIS = new BABYLON.Vector3(0, 1, 0);
        private static _Z_AXIS = new BABYLON.Vector3(0, 0, 1);

        public moveAlongX() : void { this._getMovin(PointerMoveBehavior._X_AXIS); }
        public moveAlongY() : void { this._getMovin(PointerMoveBehavior._Y_AXIS); }
        public moveAlongZ() : void { this._getMovin(PointerMoveBehavior._Z_AXIS); }

        private _getMovin(axis : BABYLON.Vector3) : void {
            this.options = { dragAxis : axis };

            // the click which will end the move
            this.attachedNode.getScene().onPointerDown = ((evt, pickResult) => {
                this.releaseDrag();
                this.attachedNode.getScene().onPointerDown = null;
            });
            this.startDrag();
        }

        /** @override */
        public get name(): string {
            return "PointerMove";
        }
    }
}

[JCPalmer]

Ok, almost 2 weeks in and Blender functionality is almost replaced directly in BJS. Implemented as a gui based editor panel, so I can pop it into any scene / page, but not have to always have it. Copied the concept from my video panel.

One thing I need, which should not be too controversial, is that some of the private methods of PointerDragBehavior, especially _startDrag, be made protected. In order to support Undo, I need to get the position just prior do starting to drag. From the above sub-class, I could hook this in _getMovin(), but that is triggered from an X, Y, or Z key press, and a simulated start drag.

I have also discovered that you can just do a normal 3D drag initiated with just a mouse-down, like in this https://www.babylonjs-playground.com/#VSURBJ#2 . Problem is I need something like

protected _startDrag(pointerId: number, fromRay?: Ray, startPickedPoint?: Vector3) {
    // record the position, for later redo
    . . .

    super._startDrag(pointerId, fromRay, startPickedPoint);
}

In order to get more complex tool scenes, lower level hooking is kind of required.