Steaming Audio into BabylonJS

Questions
1

I have a server that I can communicate with WebSockets and it delivers streaming audio to me. I am able to communicate and receive this streaming audio data, and I would like to stream it into BabylonJS. I have looked through all the current forum posts, but I haven’t managed to piece together a working solution yet.

Could anyone help me with some code that sets up the necessary audio sources, tracks, streams, etc and essentially is left with a buffer I can add data to (as it comes in from the server) and it’ll playback?

I tried adding an audio element in my HTML:

<audio id="audio"></audio>

Then over in BabylonJS:

const audio_element = document.getElementById("audio") as HTMLMediaElement;
if (audio_element && engine.getRenderingCanvas()) {
    audio_element.src = URL.createObjectURL(media_source);
    
    const media_stream = engine.getRenderingCanvas()!.captureStream();
    const sound = new Sound("Orson.wav", media_stream, scene, null, { autoplay: true, streaming: true, spatialSound: false});
    
    media_source.onsourceopen = function () {
        const source_buffer = media_source.addSourceBuffer("audio/mpeg");

        my_audio_service.onAudioData = (data) => {
            source_buffer.appendBuffer(data);   
        };
    }

The sound fails to create as it says no tracks are attached, also addSourceBuffer I initially tried to pass in wav as the MIME type, but the wav variations I tried were all rejected as being unsupported.

I feel like I must be going about this the wrong way, making it more complicated than it needs to be.

2

cc @docEdub

3

Hello scraft, and welcome to the Babylon forum!

The Babylon Sound object saying no tracks are attached is because the stream capture being created from the Babylon engine’s rendering <canvas> only has video tracks. There are no audio tracks on a <canvas> element.

For your custom streaming setup, I don’t think there’s any way to do that with a Babylon Sound object right now. I’m working on a new audio engine that will allow this, but it won’t be ready for a while, yet. In the meantime, you can use the WebAudio API’s AudioWorkletNode to stream your raw PCM .wav data.

You can access the Babylon audioContext with BABYLON.Engine.audioEngine.audioContext (which you can use to create the AudioWorkletNode), and then you can connect the AudioWorkletNode to the Babylon audio engine’s master gain node, which can be accessed with BABYLON.Engine.audioEngine.masterGain.

I hope that helps!

1 Like