Lets talk about how we represent AR scale IRL

[Calsa]

I have been playing a lot with mobile AR the last two months building some simulations and games and have noticed a bit of an issue that may be best fixed with a feature request.

Many of my projects started out in Babylon’s 3D mode, and also often fallback to 3D if AR support is not available. I also sometimes use meshes that are massive - terrains can be Kms in size , or vehicles built @ scale can be many meters tall or long. All need to remain in their original size for both use in 3D and AR, and also just for ease of the art prep pipeline.

Getting these to fit on screen as a manageable size in Babylon 3D is trivial as you can manipulate scale, camera zoom, camera perspective, ext with ease and since everything is “in screen” you don’t have to worry about real-world constraints.

But, this actually gets more tricky when in AR as all the sudden those game units (which are in meters) start taking up real-world meters or even KM of real-world space. For instance a few weeks back I had to import a 12km^2 slice of terrain . In 3D no problem seeing the whole thing in one screen by zooming out slightly. But, when I fired up AR originally I thought I broke something as I did not see it render. That is until I looked out the window of my bedroom and could see 12km of game terrain spanning out in front of me… Or today I imported a mech that at true 1unit / 1m scale is literally bigger than my house at 6 game units tall ! All I could see was its foot inside my bedroom. Everything else was out the roof! Even importing a FPS character at the standard 2m / 2 unit Babylon height makes them life size. A NPC that spawned in AR blocked my door and would not get out of my way so I could get in the hallways. Simply put, 1 game unit = 1m IRL makes things too big for most practical use in a game or sim.

The only option right now is to scale… often a TON. This leads to really small decimal numbers which are a pain to deal with. It also starts to break babylon features when you scale that much. For instance recast-detour can only build voxles down to .001 before freaking out . When I had to scale a 12km terrain down to fit a 1m game board (a reduction of 12000) I ended up with really big and borderline unusable nav mesh as each voxel is massive even at it min allowed size. PBR materials, yup those use physical rendering and are affected too. Physics packages including Havoc… yes, that’s also affected. As is… well quite a bit else I have been discovering. Even simple position.x+1 animations applied to a box when in AR means that box is jumping 1m IRL and out the roof of your house in 6-8 frames.

What may be worth discussing is adding an option in the XR experience helpers or somewhere else to under the hood to change the physical real-world representation of dimensional size of one unit in AR mode. So 1 game unit is still 1 game unit which still = 1m - keeping1m = 1 game unit is critical for fallback to 3D, and prevents breaking things relying on meters. But maybe instead of 1m IRL looking like 1m it can be remapped to visually represent 1cm or 1 mm or 1um of real-world space. Ideally this is a completely arbitrary setting to be set based on need, you could even go the opposite direction. So, its sort of like “zooming” a camera. Everything you took a photo of does not change real-world size when you zoom-out a camera, but it is smaller visually…

The importance of this or some other solution can’t be overstated for adopting AR as it opens the door wide for us to use the entirety of Babylon in games and simulations, when in many cases ‘true-to-life’ size is not quite appropriate.

I would love to hear some thoughts on this.

[mawa]

@Calsa

change the physical real-world representation of dimensional size of one unit

Yup. Totally agree. Except it would be a ‘must have’ for reliable AR, BUT also, overall, a ‘nice to have’ for any 3D scene (my opinion only).

[CodingCrusader]

Even chatGPT is enthusiastic about your idea!

GPT:
Your observation about the challenge of displaying large-scale 3D models in mobile AR is spot on. The issue you’re facing is due to the fact that AR overlays digital content on the real world, and the size and placement of the virtual objects are governed by the real-world coordinates and measurements.

The idea you propose to change the physical representation of dimensional size in AR mode is an interesting one. This would essentially allow you to scale down the virtual objects so they take up less real-world space while still retaining their size and scale in the 3D mode. This could be a useful feature for mobile AR developers who want to display large-scale 3D models without taking up too much real-world space.

One way to implement this could be to add a scaling factor to the XR experience helper or another appropriate component in Babylon. This would allow you to specify a scaling factor to be applied to the virtual objects in AR mode. For example, if you set the scaling factor to 0.01, then a virtual object that is 10 meters tall in the 3D mode would appear as only 10 centimeters tall in AR mode.

Another approach could be to add a new mode in Babylon’s XR experience helpers that would allow you to choose the scale of the virtual objects in AR mode. For instance, you could select a mode that would show the virtual objects at 1:100 scale, which would allow you to display large-scale models in a much smaller real-world space.

Both of these approaches have their pros and cons, but they would both provide a way to scale down virtual objects in AR mode without affecting their size in 3D mode. Overall, your suggestion highlights an important issue for AR developers, and it would be great to see Babylon or other AR frameworks incorporate features like this to make it easier to display large-scale 3D models in AR.

---

In my humble opinion, the following approach would also be worth considering: An AI would recognize the real objects in real time through the VR glasses and automatically transform them into meshes, so that the user only acts in the virtual world, then the different scales would be completely eliminated.

[RaananW]

Hmmm… that’s a wonderful philosophical issue
We have an open task on GitHub that could help here - World/camera scaling · Issue #12441 · BabylonJS/Babylon.js (github.com) . World scale is probably what you mean when you say

@Calsa

maybe instead of 1m IRL looking like 1m it can be remapped to visually represent 1cm or 1 mm or 1um of real-world space

But this is only a “patch”, to be hoenst. because if you load two models, one in meters, one in cm, then you can’t really define a world scale.

Another solution (not a good one, but A solution) would be to convert your models to unit cube - Mesh | Babylon.js Documentation (babylonjs.com)

This will make the mesh max 1x1x1, making it visible in AR. but this is not a viable solution as well, as you might not want exactly this kind of behavior.

There is no good solution, there are only workarounds and attempts to solve this issue.

Just my 2 cents

[mawa]

@RaananW

a wonderful philosophical issue

Glad to hear that. I share this thought.

@RaananW

if you load two models, one in meters, one in cm, then you can’t really define a world scale.

Yup. So, just my 2 cents here. I could be wrong but I have the feeling that we could may be solve this by introducing a notion around the unit cube that would not be a unit cube but rather a ‘unit base’, so to say. Similar to what is in 3D app, the ‘unit base’ would define whether the scale is defined in meters, cm, mm, feets,etc… As I said, just my two cents here thrown at you without further thoughts.

@RaananW

There is no good solution, there are only workarounds and attempts to solve this issue.

Yes,…At this point. But I think it would be worth to continue think a little about it. Eventually at some point we would find the ‘automagical’ solution that would next become just another ‘unique advantage’ of using BJS … OK, call it ‘wishful thinking’ … but at least, recognize that I’m trying to stay ‘positive’ and on the ‘bright side’ of the force

Edit: Just realize that my 2-cents are in fact a combined approach to chatGPT answers. It would in fact be a ‘scaling unit’ or ‘unit base’ (cm, m, mm…) that could next be used to further normalize imported objects through a ‘scaling factor’
Basically a blend of:

@CodingCrusader

add a scaling factor to the XR experience helper or another appropriate component in Babylon. This would allow you to specify a scaling factor to be applied to the virtual objects

@CodingCrusader

could select a mode that would show the virtual objects at 1:100 scale

except for a start, the ‘unit base’ used for normalization and as a base for scaling would be defined at scene level (in either m, cm, mm, ft or whatever)

[Calsa]

I originally thought the same thing as the GPT @CodingCrusader and @mawa . But, after literally taking two days to crack open one of my more complex projects and try to scale absolutely everything, I realized that this was a complete PITA mess to do. It also does not work. You still run into issues with physics, navmeshes, and many other non-visible things that you can’t simply scale, or only scale so much. Scale something down by a huge amount (in one scene x120000) to fit in a room completely breaks the engine in very strange ways.

This also hurts in another way as depending on scene linkage meshes may already be scaled, and parented. In complex scenes this gets tricky and becomes a pain to manage. Trust me, I had to apply three scales at once at around 120 different places in code just to try the scale option…

oof…

@RaananW 's camera scaling may work but I it depends on execution. I still question its viability though as from the PR it still sounds like its messing with the physical scale of a unit - which would throw off anything physical sized based in the engine. I’m not as talented as many here but I would think we would have to go through the whole engine and break dependency on 1U = 1M. I still would love to test it and I still wish we had that feature a few versions of Babylon ago! Sadly it got bumped to V.7. The normalization of everything to fit in 1m^3 still rescales which circles us back around to the same issue of scaling being what needs to be avoided.

So this quandary remains. Appearance of scaling without actually touching a scale value. The metaphorical analogy I keep circling back to when thinking about solutions is that of a camera. When you zoom a camera it visually shrinks without shrinking the real-world physical dimensions object. We need something like that for AR. But this is open for debate and please provide other options if you can think of them.

I actually started playing with AR camera zoom, but it gets tricky as the underlying math for the AR camera assumes 1M=1U for things when rendering and tracking placement of items in the scene. It also seems dependent on the default fixed camera aperture / zoom settings. Change any of that at your peril! I tried and instantly regretted it. That sort of math makes my head hurt but there are many more talented in that aspect than I to whom this may be a trivial change. But, even this has its weaknesses and is likely not an instant fix.

I am surprised the Microsoft core crew has not come across this given all the XR work they do. But, I guess hololense is mainly geared for industry and Microsoft exited mobile phones, so not a priority to focus on mobile AR. From what I have seen most people deving for hololense are using it for simple things that may survive scaling, or are dealing with life size items.

Anyway, I do want to keep this conversation going as there are some REALLY cool applications when you stop thinking about AR having to be 1:1 life size. There is currently no good solution for this and its a nut we should crack first. I’ll clean up and post up a few samples in the coming days as that may spark ideas.

Thoughts on moving forward? No matter how wild post them. One may work out.

[mawa]

@Calsa

keep circling back to when thinking about solutions is that of a camera

Funny, because I had suddenly the same thought this morning when having my coffee .
Not to mention all the other dependencies you mentionned above. Well, I guess I told you it was just my 2 cents, didn’t I? Obviously, I’m much better with just philosophy than Maths (so don’t count on me to find the solution). All, I can do is share ‘a feeling’ and try to ‘orientate’.

@Calsa

are some REALLY cool applications when you stop thinking about AR having to be 1:1 life size.

Yes, I also thought the same. It’s just sad the subject is so complex and deeply imbricated. Well, I guess if it would be ez, we would not be here today discussing it, would we?

[Calsa]

I was hoping I could bump this back up and see about figuring out how to make it a 7.0 feature request. The more I dive into cross-compatible Screen based 3D +XR games and experiences in Babylon, the more this becomes a very critical need given the drastic perceived scale difference between the interface environments.

I developed some hacks to get around this, but they only really work on meshes. What is really needed is a way to set another engine scale. So one game unit can mean something besides 1m. It needs to be that fundamental so everything conforms. But, that is like messing with the one true universal constant in the engine. So, would need some serious thought and planning to work across meshes, physics, lighting, materials, animations, particles, nav meshes, events, ect…

Edit I sort of wonder if you could tackle it at the other end and be better off by modifying rendering at the AR camera itself to just render things smaller when they are sent for AR integration by the device. It gets around messing with the engine itself. Something to consider, I don’t exactly know how this would work yet, but seems cleaner than touching engine constants.