I barreled in with PRs before discussing here on the forums – sorry about that. Following the feedback on my recent PRs, I’d like to explain what I’m trying to enable and discuss where it fits before attempting further contributions in the editor/playground.
My goal is to let someone create an interactive experience, or extend an existing glTF/GLB, without writing JavaScript. Longer term, we want people to do this inside a room-scale XR experience, along the lines of Dreams’ PSVR creation mode. I’m starting with incremental improvements to the flat Flow Graph Editor so the authoring, editing, and save/reopen workflows can become more durable first.
Someone was encouraging me to participate in a Hackathon-type event, and as we were riffing I came up with a fun way to gamify a high-level no-code modification to existing content. Imagine audio MadLibs: the player records a few sound effects without knowing what they’ll be used for. “Make a surprised noise.” “Make a cartoon crash.”, then those sounds get assigned to an interactive scene, and the player discovers what they’ve created. I think this also helps pressure test the KHR proposals for audio nodes as well as the implementation of KHR interactivity. I loved the no-code gamepad-only experience creation in MediaMolecules’ Dreams and LittleBigPlanet, and it’s been fun to figure out how to get closer to their Impact Sensor and Trigger Zone blocks/wiring, and how it compares to UEFN’s similar event bindings (which I had never touched before).
I thought physics interactions would be the most emergent fun, so I first built our the steps needed for Khronos’ PhysicsMath example:
- Open the existing scene.
- Select some balls and assign one sound to ball-to-ball contacts.
- Assign a different sound to ball-to-platform contacts.
- Play, change the sounds, reset, and try again.
- Save and reopen the scene with those reactions intact and working as expected.
Initially, the creator would choose existing MP3, AAC/M4A, or Ogg Opus files and preview them. Microphone recording on desktop, mobile, and headsets could come afterward.
Once this editor workflow works, I feel like I can abstract my Madlibs game on top of the editor’s plumbing under its UI. THen I can generalize the same authoring model a bit so it works for the BowShooting example: add one cue when the arrow fires and another when it lands, while keeping the existing behavior.
My first fix along the way was merged in PR #18497 (thanks!), but there’s a few other items on my list to realizing the end-to-end playable experience:
- Extend an existing interaction. A guided “Add reaction” workflow should preserve the event’s existing actions and expose the resulting wiring for further editing.
- Edit both glTF packaging forms safely. GLB and split
.gltfplus buffers/textures should retain their original node identities, hierarchy, unrelated extensions, and resources. Missing or ambiguous companion files should prompt the user to resolve them. - Make contact targeting understandable. “These balls hit each other” and “these balls hit this platform” need distinct bindings. Show the contact bounds, explain unavailable objects, and distinguish individual objects from instance groups.
- Connect events to reusable audio cues. Provide an asset chooser, auditioning, sound replacement, and predictable playback/reset behavior. Changing a sound should preserve its event binding.
- Make authoring easier to diagnose. Show which event fired and which response ran, explain failed exports, and keep playback changes separate from authored edits. I had to hit debugger breakpoints to understand why some things weren’t connecting/triggering the way I expected.
in the PRs that were closed, and the ones staged in my fork, I am thinking about keyboard, touch, screen-reader access, and localization considerations but also not trying to upend the existing structure and precedent of the existing UI. I’m thinking about this up front because I’ve learned it tees up better foundations for XR UI.
So, that’s the end-to-end player experience I’m trying to achieve, and how I’m coming at it in a way that I hope helps prove/refine GLTF for this purpose 
I don’t have strong opinions about implementation details, so I’m open to guidance on how to go about it in any future PRs (including “we don’t want to enable this”, which is fine).
Cheers!
-M