Project an image to a 3D world

Hey guys,
I have been stuck with this problem in Babylon.js for days and still couldn’t get it work.

Basically, I want to do sth like an AR application in Babylon. Take a video, and render some 3d objects inside the video. Any ideas about how to achieve this?
Most of the materials are talking about projecting 3d points to 2d images, but we want to do the opposite way.


Hi CP, welcome to the BabylonJS forum.

BabylonJS is not really designed for camera tracking and compositing. That may be a Blender/other-modeler task. Check-out this short Blender tutorial: YouTube

Let’s take a look at our previous forum… see if we can find some “augmented reality” and “mixed reality” topics. old forum search for AR

Let’s try this new forum, too. new forum search for AR

With Blender, you are working in an environment that is designed for video-rendering. BabylonJS is not really the proper tool for your project, me thinks.

That’s all I have, so far. Sorry for the possible bad news. Again, welcome. Stay tuned for more comments/ideas.

Pinging our AR mastermind @syntheticmagus

Hi chao_pan,

Awesome to hear about people trying this stuff! We have a project called BabylonAR (blog post about it here) that’s beginning to tackle issues like these. However, the project is very young (alpha), so there are still a lot of features we have yet to add to it.

Can you tell me a little bit more about your use case? Depending on the constraints on your video – for example, if you can include markers, at least in the short term – it might be possible to do what you’re hoping for with just the current capabilities. If what you need is more general case, that may veer into an extremely difficult problem called SLAM, which BabylonAR doesn’t have the capability to solve. (Not yet, anyway. :wink:)

So, in short, there may be several possibilities both inside and outside Babylon, depending on what exactly you’re trying to do. If you can share a bit more about the use case you have in mind, I’ll be happy to discuss what options might work for you. Thanks!