Proposal: per-instance bounding-info culling for thin-instance picking
Summary
The current thin-instance picking flow performs one whole-mesh broad-phase in InternalPick() and InternalMultiPick(), then iterates every thin instance and calls the picker with bounding-info checks skipped for each instance. That means dense source meshes can still pay the full submesh and triangle cost for many thin instances that an instance-level broad-phase would have rejected immediately.
This proposal updates the thin-instance path so that, when thinInstanceEnablePicking is true and rawBoundingInfo is available, the engine always runs a per-instance sphere-plus-box precheck against the source-only bounds before entering triangle intersection. There is no new property, no threshold, and no additional opt-in. If rawBoundingInfo is missing, the precheck is skipped and the current behavior is preserved, so there is no risk of false negatives from missing source bounds.
The implementation should use the existing local-ray approach already available through rayFunction, and it must mirror the intersectionThreshold handling used by AbstractMesh.intersects() for standard line meshes so thin-instance line meshes are not rejected incorrectly. One upstream caveat is that GreasedLineMesh now has a custom pick path, so it should not be treated as a drop-in match for the standard AbstractMesh.intersects() broad-phase without separate verification.
Current behavior
Today the thin-instance path in InternalPick() works like this:
- Run a whole-mesh bounding-info check once through the picker.
- If that broad-phase passes, iterate all thin-instance matrices.
- Build the combined instance world matrix from the thin-instance matrix and the mesh world matrix.
- Call the picker again for each instance, but with bounding-info checks bypassed so the code reaches
AbstractMesh.intersects()in skip-bounds mode. - Let submesh tests and triangle tests decide the hit.
That is correct, but it can do unnecessary work for thin instances that are trivially outside the ray.
Goal
Add a reject-only per-instance broad-phase before the expensive submesh and triangle walk, while preserving the existing public API, avoiding false negatives, and leaving behavior unchanged when source-only bounds are unavailable.
Updated behavior
The updated behavior is:
- Keep existing
thinInstanceEnablePickingunchanged as the only public gate. - When thin-instance picking is active and
rawBoundingInfois available, always run the per-instance broad-phase. - Do not add a new mesh property.
- Do not add a vertex-count threshold.
- Do not add an extra opt-in.
- If
rawBoundingInfois missing, skip the precheck and preserve current behavior. - Use the instance-local ray already built through
rayFunction. - Inflate the sphere and box checks with the same
intersectionThresholdlogic used byAbstractMesh.intersects()for the standard mesh and lines-mesh path. - Do not silently substitute
getRawBoundingInfo()forrawBoundingInfoin the new helper, because that upstream helper falls back to aggregated bounds when source-only bounds are missing.
Decision rationale
- a bounding sphere plus box test is extremely cheap,
- rejecting even one thin instance early usually repays the full cost of the extra prechecks,
- the downside on simple meshes is theoretical and expected to be lost in the noise compared with even a single unnecessary triangle walk,
- adding a new property or threshold would complicate the API and documentation for an optimization that should help by default.
So the plan is to treat the precheck as a low-cost reject-only optimization, not as a feature that needs new configuration.
Recommended implementation approach
Prefer local-ray plus raw bounds over world-space AABB construction
The first implementation should not build a transformed world-space AABB from 8 corners for every thin instance.
A cheaper and cleaner path already exists in the picking pipeline:
InternalPickForMesh()already uses the per-instance world matrix to derive a local ray throughrayFunction.- The thin-instance loops can derive that same local ray directly by calling
rayFunction(world, mesh.enableDistantPicking). AbstractMesh.intersects()already performs the standard sphere-plus-box broad-phase when bounds are not skipped.Mesh.prototype.thinInstanceRefreshBoundingInfo()already preserves the source mesh bounds inrawBoundingInfobefore expanding the aggregated thin-instance bounds.
So the recommended precheck is:
- Get the instance matrix exactly as today in
InternalPick()andInternalMultiPick(). - Build the same instance-local ray that the picker would use by calling
rayFunction(world, mesh.enableDistantPicking). - Resolve the same
intersectionThresholdthatAbstractMesh.intersects()would use for that mesh. - Test that local ray against the raw bounding sphere and raw bounding box.
- Only if that precheck passes, call the picker with skip-bounds mode and let triangle picking continue.
This reuses the existing picking math, avoids any per-instance corner-transform loop, and keeps the new broad-phase aligned with the normal mesh broad-phase. For the built-in picker path, it is worth structuring the helper so the same local ray can be reused for both the precheck and the subsequent mesh.intersects() call instead of recomputing it twice.
Where the checks should live
The broad-phase should live in packages/dev/core/src/Culling/ray.core.ts, directly in the thin-instance loops inside InternalPick() and InternalMultiPick().
Recommended structure:
- Keep the existing whole-mesh broad-phase exactly as it is today.
- Resolve
mesh.rawBoundingInfoonce for the mesh after the whole-mesh pass succeeds. - For each thin instance:
- apply the mesh-instance predicate,
- build the combined matrix,
- if
rawBoundingInfoexists, build the local ray and run the raw sphere-plus-box precheck, - skip triangle picking if that precheck fails,
- otherwise keep the current picker call with skip-bounds enabled.
If it helps reduce duplication, the raw-bounds precheck can be factored into a small internal helper in ray.core.ts and reused by both thin-instance loops.
Runtime condition
There is no new heuristic beyond what already exists.
Conceptually, once the code is already inside the thin-instance picking branch, the only runtime question is whether source-only bounds are available:
const rawBounds = mesh.rawBoundingInfo;
const usePerInstanceBounds = !!rawBounds;
The full decision remains:
if (mesh.hasThinInstances && mesh.thinInstanceEnablePicking) {
// existing whole-mesh broad-phase
// ...
if (rawBounds) {
// per-instance raw sphere+box precheck
}
}
If rawBoundingInfo is absent, the engine simply falls back to the current behavior for that mesh.
Raw-bounds source and fallback behavior
The new precheck must use source-only bounds, not the aggregated thin-instance bounds.
Relevant existing behavior:
Mesh.prototype.thinInstanceRefreshBoundingInfo()refreshes the base mesh bounds and copies them intorawBoundingInfobefore expanding the mesh bounds to include all thin instances.- The regular mesh
getBoundingInfo()result may already be expanded to cover all thin instances, so it is not suitable for per-instance rejection. AbstractMesh.getRawBoundingInfo()now falls back togetBoundingInfo()whenrawBoundingInfois absent, so the new helper should readmesh.rawBoundingInfodirectly and explicitly skip the precheck onnull.
Recommended fallback rules:
- If source raw bounds are available, use them.
- If source raw bounds are missing, do not try to derive per-instance culling from the aggregated mesh bounds.
- Instead, skip the new precheck and preserve current behavior for that mesh.
That avoids false negatives and avoids using overly large aggregated bounds that would provide little or no benefit.
Broad-phase details
The new per-instance precheck should mirror the existing mesh broad-phase in AbstractMesh.intersects() for the normal mesh and lines-mesh path:
- use the same
intersectionThresholdlogic used there today, - test the local ray against the raw bounding sphere first,
- then test against the raw bounding box,
- continue to triangle picking only if both pass.
Conceptually:
const threshold =
className === "InstancedLinesMesh" || className === "LinesMesh"
? (mesh as any).intersectionThreshold
: 0;
if (!localRay.intersectsSphere(rawBounds.boundingSphere, threshold) || !localRay.intersectsBox(rawBounds.boundingBox, threshold)) {
continue;
}
The important correctness detail is that LinesMesh and InstancedLinesMesh need the same threshold inflation here that they already get in AbstractMesh.intersects(). Without that, line-mesh thin instances could be rejected before their actual picking path has a chance to run.
GreasedLineMesh is not on the standard broad-phase path
GreasedLineMesh now overrides intersects() and routes picking through width-aware segment tests in findAllIntersections(). Its broad-phase behavior is therefore not the same as the standard AbstractMesh.intersects() sphere-plus-box path:
- its
onlyBoundingInfocheck uses a sphere test, not the standard sphere-plus-box pair, - its actual segment precision scales with line width and per-segment width, not just the plain
intersectionThreshold.
Because of that, the proposal should not treat GreasedLineMesh as automatically covered by the generic raw sphere-plus-box helper. The conservative implementation choices are:
- either limit the first pass of this optimization to meshes that follow the standard
AbstractMesh.intersects()broad-phase, - or add a GreasedLine-specific precheck only if it is proven to match the custom width-aware picking semantics without false negatives.
Behavior and compatibility requirements
Preserve current public API
This proposal intentionally introduces no new public property and no new scene-setting behavior. Existing thinInstanceEnablePicking semantics remain unchanged.
Preserve onlyBoundingInfo
The current thin-instance path returns early after the whole-mesh broad-phase when only bounding info is requested. The proposal keeps that unchanged. The new per-instance precheck is not needed in that path.
Preserve fastCheck
Once a valid pick is found, fast mode should still return immediately. The precheck only reduces the number of expensive instance tests that are attempted before that point.
Preserve predicate behavior
The mesh-instance predicate currently runs before each thin-instance pick attempt. That ordering must remain unchanged.
Preserve pick data
The following result fields must remain correct:
- thin instance index,
- picked point,
- distance,
- submesh id,
- picked mesh reference.
Custom picker compatibility
PickingCustomization can override the picker used for meshes. The new per-instance broad-phase should still run before invoking that picker when rawBoundingInfo is available, because it is a reject-only optimization. If raw bounds are not available, the custom picker should continue to see the current behavior unchanged.
Public API and serialization
No new public API is proposed.
That means:
- no new property on
Mesh, - no new serialization field in
packages/dev/core/src/Meshes/mesh.ts, - no parser changes for thin-instance scene data,
- no scene round-trip changes beyond existing behavior.
Validation plan
Correctness tests
Add tests that compare current behavior and the new behavior in the cases where the precheck should and should not apply.
Coverage should include:
- thin instances translated away from the ray,
- thin instances rotated and non-uniformly scaled,
- negative scaling or mirrored transforms,
- dense meshes and low-vertex meshes,
- hit and miss parity,
- preservation of thin-instance indices,
- preservation of
fastCheck, - preservation of
onlyBoundingInfo, - fallback behavior when
rawBoundingInfois missing, LinesMeshorInstancedLinesMeshwith nonzerointersectionThreshold,- any
GreasedLineMeshthin-instance case that is either explicitly excluded from the new helper or covered by a separate verified implementation.
The critical guarantee is no false negatives when the new precheck is active, including the standard line-mesh threshold case and any explicit GreasedLine handling.
Performance validation
Benchmark at least these scenarios:
- Dense source mesh, many thin instances, low hit ratio.
- Dense source mesh, many thin instances, high hit ratio.
- Low-vertex source mesh with a small thin-instance count.
- Low-vertex source mesh with many thin instances, to confirm the always-on precheck is still neutral or beneficial in practice.
- A case where
rawBoundingInfois absent, to confirm behavior and cost stay aligned with the current path.
Measure time spent in scene picking paths that flow through InternalPick() and InternalMultiPick().
Non-goals
This proposal does not aim to:
- add a new per-mesh thin-instance picking property,
- add a vertex-count threshold or heuristic gate,
- introduce a global static switch for the optimization,
- add per-instance cached bounding boxes or spheres,
- redesign non-thin-instance picking in
AbstractMesh.intersects(), - replace the current triangle picking path with a new acceleration structure.
Risks
Key risks for implementation are:
- false negatives if the precheck accidentally uses aggregated thin-instance bounds instead of
rawBoundingInfo, - false negatives for
LinesMeshorInstancedLinesMeshif theintersectionThresholdinflation does not matchAbstractMesh.intersects(), - false negatives for
GreasedLineMeshif it is forced through the generic helper without matching its width-aware pick semantics, - accidental use of
AbstractMesh.getRawBoundingInfo()in the helper, which would silently convert the intendedrawBoundingInfo-missing fallback into an aggregated-bounds precheck, - stale source bounds if updates around
Mesh.prototype.thinInstanceRefreshBoundingInfo()are bypassed or not synchronized with source geometry changes, - compatibility surprises for custom pickers wired through
PickingCustomizationif the reject-only helper changes ordering incorrectly, - small regressions in pathological all-hit or tiny-mesh scenes where the extra broad-phase prunes very little work.
Performance impact
Expected CPU impact:
- positive for dense source meshes with many thin instances and low hit ratios, because fewer instances reach the expensive submesh and triangle tests,
- positive or neutral for many practical scenes because the added sphere-plus-box test is very cheap and rejecting even one instance tends to pay it back,
- theoretically slightly negative in tiny-mesh or all-hit cases, but expected to be noise compared with even a single unnecessary triangle walk,
- unchanged when
rawBoundingInfois missing, - unchanged for
onlyBoundingInfo, because that path already exits after the whole-mesh broad-phase.
Memory impact
Expected memory impact:
- near-zero steady-state increase because the implementation reuses existing
rawBoundingInfo, - no new property on
Mesh, - no new per-instance arrays in thin-instance storage,
- no meaningful hot-path allocation increase if the implementation reuses the existing temporary math objects already used by the picking flow in
packages/dev/core/src/Culling/ray.core.ts.
Mermaid overview
flowchart TD
A[Whole mesh bounds pass]
B{Only bounding info}
C[Thin instance loop]
D{Predicate passes}
E{Raw bounds available}
F[Local ray plus thresholded sphere and box precheck]
G[Skip instance]
H[Triangle picker with skip bounds]
I[Update picking result]
A --> B
B -- yes --> I
B -- no --> C
C --> D
D -- no --> G
D -- yes --> E
E -- no --> H
E -- yes --> F
F -- fail --> G
F -- pass --> H
H --> I
Implementation handoff
Implementation mode should execute the following steps:
- Add a small internal helper in
packages/dev/core/src/Culling/ray.core.tsthat tests a thin instance againstmesh.rawBoundingInfousing the instance-local ray and the sameintersectionThresholdrule asAbstractMesh.intersects()for the standard mesh and lines-mesh path. - Apply that helper in both
InternalPick()andInternalMultiPick()after the whole-mesh broad-phase passes and after the per-instance predicate check. - On a failed raw-bounds precheck, skip triangle picking for that thin instance.
- On a passed precheck, keep the current picker call with skip-bounds enabled.
- If
rawBoundingInfois missing, preserve the current behavior with no extra rejection step. - Keep current behavior for
onlyBoundingInfo,fastCheck, predicates, custom pickers, and thin-instance result data. - Add tests for hit parity, false-negative prevention, direct-
rawBoundingInfofallback behavior, and standard line-mesh threshold inflation. - Decide explicitly whether
GreasedLineMeshis excluded from the first implementation or handled by a separate verified precheck. - Benchmark dense and simple cases to validate the always-on choice.
Alternatives
- Per-instance Octree, updated only on thinInstanceSetBuffer(“matrix”) or equivalent, costing extra memory usage and update cpu overhead
- Keep it as-it, thin instance picking is already an opt-in feature, users should take their own risk

