Edit: Just thinking of it right now. Now that I understand better where you’re heading.
Ever thought of using Houdini for the bake and the backend? May be check this out and cc @br-matt might be able to give you some hints on the process they used for this awesome project, will you? thx a lot 