How to Stream Audio into the Engine

Questions
1

I am working on a peer to peer application, and one of the things I would like to be able to do is have a user share audio with the others. Currently looking to do this with webtorrents.

Now, with a webtorrent I can get a readable stream which gives me chunks of data in the appropriate order, with each chunk being a node Buffer. And each chunk of data I can turn into an arraybuffer on either on their own, or all together to get the completed audio file. I’ve not figured out how to convert a node Buffer into a mediastream either.

My problem comes when I want to consume these chunks. After the audio from a chunk has played, I’d like to discard the chunk so I’m not just holding the audio in memory. But, from what I can tell that’s easier said than done.

The sound constructor can accept a url, an arraybuffer, or a mediastream.

I could do a blob uri for the url, but that’d require having the entire audio clip already loaded to do that which defeats the purpose. An arraybuffer is slightly better. Things can be added into it over time, so I don’t need everything loaded already to run it. But to my knowledge, it doesn’t consume and discard the data as its used. It holds onto it. Mediastreams are a bit more of an enigma to me, they look like they are used specifically for pulling media from a user’s camera and microphone which isn’t what I’d like to do. And I’m not even sure if they would do what I’m wanting either because I don’t know if they discard media, or hold onto it.

My current thought is to create several sound objects in babylon, each of which holds a chunk of the audio, and play them back to back. However this seems clunky, and I’m not sure it could work.

Some additional notes. The audio files could be of variable length, anywhere from a few seconds long to hours. Also, I’m not tossing the chunks completely after they’ve been used. That would defeat the purpose of using webtorrent. I just want to pull it out of memory.

So with this in mind, is there a way to stream music into the babylon engine? And what about other media, like videos?

2

Hello @missionz3r0 - and welcome to the Forum!

I’ll start by sharing some general information, before some more expert folks jump-in to help you:

1 Like
3

Yep, read over those and I don’t know how many other docs while working on this project.

Shows how to stream things in, but not out. If I’m playing a 2 hour long piece of audio, and I’m 70 minutes in, I don’t really need the last 60+ minutes loaded into memory right at that moment.

4

That will probably require a server capable of streaming music directly. From the client standpoint, if you give a url to a stream then it will work transparently

5

I can tell you that a server isn’t needed in this case. I’ve already tested uploading and sending a file from client machine to client machine. I can also pass the file into Babylon, however it requires either putting the whole file into it, or eventually putting the whole file in. But what I’m trying to do is stream in some while streaming out parts already used.

I’m starting to think I could just split the file into something like 30 second chunks, and playing them back to back, destroying the already played audio as I go.

6

probably the best option if you do not want to dig into RTMP protocol

7

Use <audio /> if that is a browser based app. Set audio.src to object url of a MediaSource, then add a SourceBuffer to mediasource, then stream data into source buffer. To get MediaStream from audio, use audio.captureStream().

const media_source = new MediaSource()

audio.src = URL.createObjectURL(media_source) 

const media_stream = audio.captureStream() 
const sound = new BABYLON.Sound(.., media_stream, ..)

media_source.onsourceopen = function () {
   const source_buffer = media_source.addSourceBuffer(..)
   // ...
   source_buffer.appendBuffer(/* buffer here */)   
}
2 Likes