Move 3D obj File with XYZ co-ordinates from video

Questions
1

i am trying to show a 3D obj file on a video by tracking the objects inside the video, And i am getting a XYZ array of points from video element.

var [x, y, z] = [259.03500787877914, 213.21602616932168, 6.576560020446777];

I am using the Babylon.js VideoTexture element, and passing the video element to it, and also adding the obj file like this,

var object;
BABYLON.SceneLoader.ImportMeshAsync("",  '/', 'file1.obj', scene).then(function (result) {
    object = result.meshes[0];
});

Now, i need to move the “object” with the x,y,z co-ordinates. But if i create a Vector with xyz. its not working,

i am using the below code to move the “object”

var vect = new BABYLON.Vector3(x, y, z);
object.position = vect;

Or

object.position.x = x;
object.position.y = y;
object.position.z = z;

both the codes not working, the object not moving properly on the canvas. its going, outside the Canvas element.

2

Are you setting the object position inside the callback, meaning just after the line object = result.meshes[0];?

Also, I think a PG would ease looking for the problem.

3

Hi Evgeni, Yes, i am setting the position inside the callback. i tried outside the callback, but the position is not working properly.

the XYZ are from Tensorflow Handtrack.js co-ordinates,

4

I had a quick look at the handtrack.js lib (Handtrack.js: tracking hand interactions in the browser using Tensorflow.js and 3 lines of code — The TensorFlow Blog - nice lib by the way).

As I understand it, you should get a bounding box, maybe a point (the center of the bounding box), but in 2D. I don’t see how you can get a z coordinate?

I think you will need to setup either a PG or an external project (github?) for us to be able to help further, as for the time being I don’t think I understand fully what you are doing.

5

Oh Sorry, my bad, its HandPose.js, it gives a XYZ for each point.

landmarks: [ // The 3D coordinates of each hand landmark.
[472.52, 298.59, 0.00],
[412.80, 315.64, -6.18],
…
],

I want to add a Ring inside a Finger. The Ring is a Obj file + mtl file.
and when i am passing these XYZ into a 3D obj file as a Babylon.Vector3 and also if passing the X,Y, Z to the position of the meshes, the obj moving out of the screen and not displaying at the specified point(XYZ).

6

Unfortunately, the z coordinate won’t be of much use to you, as the video is a 2D image: you won’t be able to have the ring partially hidden by some fingers because there is no depth in the video image.

The X/Y coordinate (screen coordinates) will allow you to draw the ring at the right position but not to hide a part of it.

7

But are’nt those having XYZ co-ordinates?? So, z also included, which is a depth, right.

and do you know, how i can use those XYZ co-ordinates to view the 3D object on a finger.?

8

The picture is a 2D one, there’s no depth / Z, only X/Y coordinates. The Z coordinates that the lib is generating may help to sort the 21 points between each other, but relatively to the 2D picture they have no meaning (they would have if the lib would generate a Z coordinate for all the points of the 2D picture, which it does not).

So, in my understanding, the best you can do is displaying the ring at one of the 21 points the lib gives you, but in 2D, not in 3D.

9

Okay Thanks, But Do you know, how can we show the XY points only, I am getting points like this,

[259.03500787877914, 213.21602616932168, 6.576560020446777]

but, if i am passing them directly, the 3D obj not moving properly.

10

I guess the XY coordinates are relative to the picture, but without a repro it’s hard to tell more.

11

okay, no problem. thanks