Hello all! I think I have improved the integration between babylon and cesium, and wanted to share it with the forum and see what everyone thought. 
Controls are WASD/qe and IJKL/uo or “5.5 DoF”
There were a few hurdles to get over, and ill explain them here. Please tell me where I can improve the thinking here if you see a chance.
Cesium normally needs to run almost in what I would call parallel to babylon, using two webgl context, and there were issues with getting it to function as more than a passive display. I could find no way to make it intractable, such that I could click in babylon and select a tile mesh.
My thought was to directly leverage the 3dtileset class from cesium, and sync the camera mechanics such that they fed into what cesium expects. This bypasses the Viewer and all the other cesium bits. That is mostly what I ended up with, but there were some cool talking points along the way…
Cesium’s 3dtileset constructor annoyingly forces the creation of a DynamicEnvironmentMapManager, which wants a webgl context to operate. It turns out, simply removing this function from the 3dtileset constructor removes the need for the webgl context, and just works without it. I’m not too sure what the dynamicenvironmentmapmanager does, but I’m assuming it has something to do with day/night cycles in cesium. With that out of the way, no more webgl context needed! We are directly using the 3dtileset class, so we also don’t need a cesium Viewer and all that.
Camera syncing was fairly straightforward, after figuring out the coordinate system conversions. I was stuck for months on the coordinate conversions, and rotation conversions! Creating a cesium camera powered by babylon camera numbers was kinda fun.
Cesium’s real state-sharing mechanism is a ‘framestate’ which has all kinds of things in it like the camera, culling volumes, momentum, and lastupdate timestamps. After getting that in place, I needed to add the update call for cesium’s loop into the babylon render loop so cesium’s calculations can run.
Cesium has a factory pattern that lets you override the bits that load tile models (gltf), and so I made a simple override to load/show unload/hide the models in babylon. This is one part I would specifically like to hear some insights and feedback on. I’m not sure the approach I’m taking here allows cesium the maximum ability to interface to babylon.
Cesium’s drawcommand list mechanics make rendering in babylon a bit tedious for me. Cesium only adds the draw commands when it wants to show the tile. If the draw command for a tile is not included, cesium knows not to show it. That’s fairly simple until it tries to interface with babylon, which is designed to show the mesh every frame until you tell it not to. I end up having to setEnable(false) on all the tile meshes, each frame, and then setEnable(true) on the ones cesium selects. Does anyone know how to instruct babylon to draw the mesh/texture object, but only for that single frame?
Thanks to useLargeWorldRendering for making all this not look terrible! 
New: Large World Rendering! (experimental)
Source code:
Optional dynamic environment map manager by techtruth · Pull Request #13077 · CesiumGS/cesium · GitHub – PR to cesium asking for an optional condition exposed to create the DynamicEnvironmentMapManager. Disabling this allows cesium to run without a Viewer, or attaching itself to a canvas/webgl context.
GitHub - techtruth/cesium_in_babylon: Babylon rendering with cesium data in one WebGL context. (No second canvas) – PoC demo exists here in various branches.