Label occlusion - babylon.js vs Sketchfab?

Questions
1

(Sorry for all the questions today, this is the last one for a while! :wink: )

I’ve looked back through quite a few threads on here about GUI / label occlusion, and seen a bunch of different approaches e.g. raycasting - but they all tend to have quite big performance issues (especially on mobile) as soon as you’re looking at more than a couple of low-poly models.

But I was looking at one of many labelled models on Sketchfab and wondering if what they achieve might be possible in babylon somehow. Their method seems to be highly performant (which raycasting with lots of / high poly meshes and more than a couple of labels etc. is definitely not :sweat_smile: ), and they also seem to have very precise occlusion - the labels don’t just change transparency based on one point on them, but gradually are properly occluded “pixel by pixel”. It almost suggests they’re using some kind of shader approach. It also works with no visible drop in framerate on my phone, while raycasting in babylon made it virtually unusable.

Basically what I’m asking is - does anyone have any pointers of what I could look into more to get similarly highly performant labels working here? I don’t really care too much about the gradual partial occlusion for the labels - just basic occlusion would be good if it were performant. If Sketchfab does it, surely with all the great minds working on babylon.js it must be possible here as well?

Any recommendations / pointers greatly appreciated, thank you!

2

This is quite easy to achieve.

You have two options (you always have more :wink: ) but for now 2:
Create the labels in your 3D modelling software and make them part of the model. Set billboard mode for the labels after the model was loaded.

or

You can create the labels dynamically as planes in 3D in billboard mode. Each plane has a dynamically generated texture with the label/number.

Since the labels are rendered in 3D they will be occluded automatically.

3

Thanks for the ideas!

They need to be dynamic, so unfortunately can’t be added as part of the model.
I did look at planes + billboard mode - the issue though is that that will totally occlude them and in some cases cause clipping issues with geometry, as some of them will be in quite tight spaces. I need them to be visible as long as the TransformNode they’re at is, and semi-transparent if their position is occluded… It’s a more difficult problem than basic occlusion unfortunately :slightly_frowning_face:

4

Ok, let’s create a playground :wink:

I’ll create the basic stuff and we can fine tune it together.

2 Likes
5

Let’s start with this:

6

Thank you so much for putting in time to help me! I really appreciate it.
Here’s a playground to show what I was doing before - works fine with a sphere but the performance is terrible with a 90k vertex mesh :sweat_smile:

I’ll take a look now at your playground!

7

We will need some custom shaders here, I guess.

Do you have Spector.js?

With Spector we can peek into the rendering process of the sketchfab model. I already did and oh man there is a lot stuff happening!

8

Nope, never heard of it. I can download it though - being able to look at the rendering process of e.g. sketchfab models sounds useful.

In the playground I linked before you can see that it’s replicating the “ideal behaviour” of sketchfab (geometry occlusion, semi-transparent when occluded), but more simplified of course! So I guess the idea would be to try and replicate something similar but in shaders as you say, so that it doesn’t lose loads of performance to Rays…

EDIT: Installed Spector.js and wow, it does give an impressively detailed overview of the rendering!

1 Like
9

If you need this simplified version I suggest to try GPU picking to boost the performance.

10

I did consider that - unfortunately though even with the fairly simple example in the playground linked in the GPU picking thread, the framerate halves on mobile. So I can’t imagine it would be any better with 90-120k vertex models with 5-30 labels… :cry:

11

Try to load your model here instead of the sphere.

1 Like
12

image
image556×642 42.7 KB

1 Like
13

It’s certainly a lot better than CPU rays. The issue is that even with a model with “only” 200k active faces and 5 labels, on the mid-tier mobile throttling in devtools it’s jumping between 10-30 fps, with sudden drops to 2-4fps every now and then. Without the picking it runs at 40 - 60 fps fairly consistently it seems, so it’s knocking a lot of frames off… I guess that could be partially mitigated by increasing the updateInterval though at the cost of slower updates (which starts becoming noticeable).
And more importantly, seems to be incorrectly checking occlusion :thinking:
Side view:
image
Around to the right (so the label is in front):
image

EDIT: New playground version just with the multiple markers and file loading, although of course the file won’t load as it’s a local one

EDIT 2: Ah - presumably it doesn’t work for some of the markers (e.g. the one in the screenshots) because it’s just checking if there’s a mesh at the coordinates - without taking into account whether that’s in front or behind of the marker? It seems to work sometimes though (image below), so maybe the picker doesn’t take into account the backface of meshes? That would probably explain it.

image
image399×563 76.8 KB

14

It’s still running like crazy on my iPhone 12.

However I could imagine to have a gpuPicker.pickMultipleAsync(xy: Vector2[] | number[] | whatever): IGPUPickResult[] @Deltakosh :wink:

15

Hmm, curious. Tomorrow morning I’ll try getting my project working with the GPU picking and try it out on a couple of phones - it unfortunately needs to run well-ish on 5-7 year old low-range phones as well as it’s intended for classes of students, who don’t alway have lots of money. So it means performance is quite important :upside_down_face:

I’ll also take a close look in the morning to see if I can work out how to sort out the GPU picker not caring if the picked mesh is in front or behind the label…

Thanks so much for your help - I definitely feel closer to a solution than I was before! Apologies for taking up so much of your time. Let me know if you have a Ko-Fi or PayPal or something I can “buy you a coffee” on as thanks :smiley:

1 Like
16

image
image966×556 47.5 KB

Don’t even think about it :wink:

2 Likes
17

@Deltakosh @sebavan if you guys decide we don’t need this I’ll simply discard the PR - no worries!

Still in DRAFT - needs some fixing.

1 Like
18

I like it but I would love to see if we can factor the code more as most of it is identical

19

Okay - I’ve been trying to fix the GPU Picking method we got to but unfortunately am struggling to understand how it could work properly…

As far as I understand, what this code is doing

Code 
const transformMatrix = scene.getTransformMatrix();
const viewport = camera.viewport.toGlobal(engine.getRenderWidth(), engine.getRenderHeight());
const coords = BABYLON.Vector3.Project(marker.node.getAbsolutePosition(), BABYLON.Matrix.Identity(), transformMatrix, viewport);

picker.pickAsync(coords.x, coords.y).then(pi => {
    if (pi?.mesh) {
        marker.circle.alpha = 0.3;
    } else {
        marker.circle.alpha = 1;
    }
})

is projecting the 3D position of the GUI element (or rather its relevant TransformNode) into 2D, and then in a specific 2D render pass the GPU picker checks whether any of the meshes that were passed to it overlap that 2D position (having rendered them out in specific colours in that pass).

If that’s correct, then I’m struggling to understand how, from that, it would be possible to determine whether the picked mesh is in front or behind the 3D position of the GUI’s TransformNode. Surely, as it’s happening in 2D, and the TransformNode doesn’t have anything rendered for the picker to see occluding a mesh behind it, it wouldn’t be possible? Leading to e.g. it thinking it’s occluded here even though the label is actually in front of the mesh:

image
image1484×645 70.9 KB

That doesn’t however then explain how label M1 is “not occluded” here though. If my understanding of the GPU picker is correct then I’d expect this one to also be “occluded”, as the mesh is rendered behind it. Might it be to do with the normals or something? :thinking:

image
image997×549 134 KB

Am I missing something obvious about the GPUPicker? :sweat_smile:

20

Sure, no worries buddy. I wanted to share the snapshot build with someone :sunglasses: that’s why I created the PR and marked as Draft.

2 Likes