AudioEngineV2 - How to attach sound to mesh in a new way

Hello @docEdub

Previously, in AudioEngine, I used to create a sound with new Sound(...), and it had the attachToMesh function, which allowed me to attach the sound to a TransformNode so it would follow an object in the scene.

Now, I’m using CreateSoundAsync, but it returns a Promise<StaticSound>. The problem is that StaticSound doesn’t have a setPosition or attachToMesh method, and I need the sound to work correctly as spatial sound and follow an object.

How can I now set the position of the sound source in space for StaticSound? Is there a way to get Sound instead of StaticSound?

I’d really appreciate any help! And sorry if I missed something in the documentation or the forum.

Thanks!

2 Likes

Thanks for using the new audio engine already! It’s not done and there’s no docs, yet, so using it now will be a little bumpy, but here’s some early-adopter tips …

Sound spatialization in the new audio engine is now done using the spatial property. See the API docs for the StaticSound.spatial property for more info.

An attach() function has been added for transform nodes, meshes and other graphics objects, and will be available in the next minor release.

I’ll add the remaining audio features this week and start adding docs, soon.

4 Likes

Thank you for the quick response!

I always try to stay on the cutting edge and update to the latest versions of Babylon.js. The recent update affected the sound system in my project, and I realized it’s time to switch to the new audio engine. I definitely don’t want to roll back to the legacy solution, so I’m looking forward to the updates to take full advantage of the latest features.

Thank you for all your hard work!

1 Like

Can you tell us how? While audio engine 2 is the new system, our promise to back compat remains.

Sure! The last version where everything worked fine was 7.51.3. After upgrading to 7.52.2, I noticed that sounds were no longer audible. There were no errors in the console, so I suspect the issue might be on my side.

I apologize for not providing a Playground example. My project is quite large, and it will take some time to extract a minimal reproducible case. However, I can show you the class I use for sound playback, which stopped working after the update. Again, it’s entirely possible that the issue is due to something on my end.

Here is a standalone class I wrote as an abstraction layer for handling sounds:

import { AbstractMesh, AssetsManager, Scene, Sound } from 'babylonjs'

class SoundPlayer {
  private readonly baseUrl: string = '/sounds/loader/'
  private readonly scene: Scene
  private assetsManager: AssetsManager
  private mesh: AbstractMesh | null = null
  sounds: { [key: string]: Sound[] } = {}
  volume: number

  constructor(scene: Scene, baseUrl: string, mesh: AbstractMesh | null, volume: number = 1) {
    this.scene = scene
    this.assetsManager = new AssetsManager(this.scene)
    this.baseUrl = baseUrl
    this.mesh = mesh
    this.volume = volume
  }

  setVolume(volume: number) {
    Object.values(this.sounds).forEach((soundArray) => {
      soundArray.forEach((sound) => {
        if (sound.isPlaying) {
          if (this.volume > 0) sound.setVolume(this.volume)
        }
      })
    })
    this.volume = volume
  }

  setMesh(mesh: AbstractMesh | null) {
    this.mesh = mesh
  }

  playSound(name: string, volume: number = 1.0, loop: boolean = false, fade: boolean = false, playbackRate: number = 1): Sound | undefined {
    if (name in this.sounds) {
      const randomIndex = Math.floor(Math.random() * this.sounds[name].length)
      return this.playSoundFile(this.sounds[name][randomIndex], volume, loop, fade, playbackRate)
    } else {
      console.error(`Sound play '${name}' is not defined.`)
    }
  }

  stopSoundFadeOut(name: string, fadeout: boolean = true): void {
    if (name in this.sounds) {
      this.sounds[name].forEach((sound: Sound) => {
        if (fadeout) {
          const fadeDuration = 3000 // Adjust fade duration as needed
          const intervalDuration = 50 // Adjust interval duration as needed

          const initialVolume = sound.getVolume()
          let currentVolume = initialVolume

          const interval = setInterval(() => {
            currentVolume -= initialVolume / (fadeDuration / intervalDuration)
            sound.setVolume(Math.max(currentVolume, 0))

            if (currentVolume <= 0) {
              clearInterval(interval)
              sound.stop()
            }
          }, intervalDuration)
        } else {
          sound.stop()
        }
      })
    } else {
      console.error(`Sound '${name}' is not defined.`)
    }
  }

  getSound(name: string): Sound | undefined {
    if (name in this.sounds) {
      const sounds = this.sounds[name]
      if (sounds.length > 0) {
        return sounds[0]
      } else {
        console.error(`No sounds available for '${name}'.`)
        return undefined
      }
    } else {
      console.error(`Sound '${name}' is not defined.`)
      return undefined
    }
  }

  stopSound(name: string) {
    if (name in this.sounds) {
      this.sounds[name].forEach((sound: Sound) => {
        sound.stop()
      })
    } else {
      console.error(`Sound '${name}' is not defined.`)
    }
  }

  playSoundFile(
    sound: Sound,
    volume: number = 1.0,
    loop: boolean = false,
    fade: boolean = false,
    playbackRate: number = 1,
  ): Sound | undefined {
    sound.setVolume(0)
    sound.loop = loop
    sound.setPlaybackRate(playbackRate)
    sound.autoplay = true
    if (this.mesh) {
      sound.spatialSound = true
      sound.setPosition(this.mesh.position)
      if (this.scene.activeCamera) {
        sound.setLocalDirectionToMesh(this.scene.activeCamera.position)
      }
    } else {
      sound.spatialSound = false
    }

    if (fade) {
      const fadeDuration = 1 // Adjust fade duration as needed
      sound.play()
      sound.setVolume(0, fadeDuration / 2) // Start with silence and gradually increase volume
      setTimeout(() => {
        sound.setVolume(volume * this.volume, fadeDuration / 2) // Gradually increase volume to target
      }, fadeDuration / 2)
      return
    }
    sound.setVolume(volume * this.volume)
    sound.play()

    return sound
  }

  async fetchUrls(urls: Record<string, string[]>) {
    try {
      for (const action in urls) {
        if (Object.prototype.hasOwnProperty.call(urls, action)) {
          const actionKey = action as keyof typeof urls
          const soundTasks: Promise<Sound>[] = []
          urls[actionKey].forEach((url) => {
            const soundTask = this.assetsManager.addBinaryFileTask(`${actionKey}-${url}`, this.baseUrl + url)
            soundTasks.push(
              new Promise((resolve, reject) => {
                soundTask.onSuccess = (task) => {
                  const sound = new Sound(url, task.data as ArrayBuffer, this.scene)
                  resolve(sound)
                }
                soundTask.onError = (_, e) => {
                  reject(`Failed to load sound '${url}': ${e}`)
                }
              }),
            )
          })
          Promise.all(soundTasks)
            .then((sounds) => {
              this.sounds[actionKey] = sounds
            })
            .catch((error) => {
              console.error(error)
            })
        }
      }
      return await this.assetsManager.loadAsync()
    } catch (error) {
      console.error('Failed to load sounds:', error)
      throw error
    }
  }
}

export default SoundPlayer

I will dig it more.

Appreciated!

To pave the way for the new audio engine, the old audio engine does not get created automatically anymore when the graphics Engine is initialized. You can enable the old audio engine by setting the audioEngine: true option in the Engine constructor, like this:

const engine = Engine(canvas, true, { audioEngine: true }, true);
2 Likes

Yo @docEdub … I am having trouble with spatial sound and moving around the map. I have a driving game and attach the spatial to that node, but i can only here it when i am close to position 0,0,0.

I have SpatialAutoUpdate = true.

Can you please show example of spatial audio moving around as attached to a mesh ?

Unfortunately, this doesn’t help. I keep looking for the cause. :face_with_monocle:

1 Like

Please create a new thread so we keep this thread focused on the original issue, and (as always) please create a playground showing your issue so it can be diagnosed more quickly and accurately. Thanks for the help!

I dont really have a playground that shows my exact usage. I am using in the Babylon Toolkit.

I was hoping you had a playground that showed how to use the new Audio Engine V2 with spatial support and attached to a mesh that may be far away.

I am not sure the proper setup for spatial and i assume you simply attach to the mesh.

I will make another thread and try to setup something, but it wont be my exact usage, but hopefully enough to figure out the proper setup and usage

1 Like

@docEdub Andy, I need your help reproducing a bug. How can I disable the audioEngine in the Playground by setting it to false to ensure that sounds are not playing because of it?

I can’t dispose of and recreate the engine in the Playground.

Here’s my test case:

Thanks for reporting this!

For backward compatibility we’re enabling the old audio engine in playgrounds that contain BABYLON.Sound, even if the engine’s audioEngine option is set to false like in this playground: https://playground.babylonjs.com/#QARKUZ#5.

PR 16273 should fix the issue. When that PR is merged.


I looked into why the Sound is failing silently, and it’s because when the Sound constructor sees there’s no audio engine, it assumes the browser doesn’t support WebAudio (which was fairly common way back when the original audio engine was built).

Setting the engine’s audioEngine option to true should get it working, though, so I’m not sure if this is an accurate representation of your issue.

1 Like

Hmmm… I am setting audioEngine: true but i get no sound when (Even from console)

        var sound = new BABYLON.Sound(
            "loading",
            "https://raw.githubusercontent.com/onekit/gardener/master/public/sounds/loader/loading.ogg",
            scene,
            null,
            { autoplay: true }
        );

But it does work in Playground…

And my engine does specify audio engine property:

var engine = new BABYLON.Engine(divcvs, antialias, { audioEngine: true, powerPreference: cpuPerformanceProfile, preserveDrawingBuffer: preserveDrawBuffer, stencil: useStencilBuffer, premultipliedAlpha: false, doNotHandleContextLost: true }, false);

Could be the browser’s audio context is locked when the sound is created. That will cause the sound to not play if it is not looped. Try setting the sound’s loop option to true to see if that’s the problem. Otherwise, post a complete code example in the playground or a github repo, please.

Thanks.

I’ve been struggling for a day now to get sound working and been working through the documentation and now have come over here to discover that all the documentation basically doesn’t work with my version of BabylonJS (i.e. v7.52.2).

The example of unlocking audio simply fails with unlock is not a function. In the playground there was the first reference to AudioEngineV2 - are there any examples using the new AudioEngine? I haven’t found any…

Particularly frustrating is that playground examples work in the same browser, in the same WebXR (Quest3s) environment yet my code does not work. And then I read that legacy code is automagically supported in the playground yet not by default using the new version of BabylonJS.

The audioEngine: true option on the engine simply doesn’t not work for me.

So my suggestion would be that there is a big red (ideally flashing!) warning in the audio documentation pointing out that it only applies to the old audioEngine and that it could be activated using the audioEngine option.

Sorry had to get that off my chest because even though video works, sound is a completely wash out :frowning:

But still enjoying BabylonJS :slight_smile:

Yo @gorenje … I started another thread about audioEngine: true not working

1 Like

I found the issue and fixed it with PR 16278. The audioEngine: true option should work again when that PR gets merged and released.

Thanks again for all your excellent work @docEdub

1 Like

Could the import of the audioEngine file also be missing?

2 Likes