Gizmo issue in multi-canvas scene

Hello,

I encounter weird behaviour of gizmos and wonder if it’s a bug or an issue with my code.

Situation:

  • I use ‘registerView’ feature of engine to use single scene and multiple canvases.
  • Each view is registered with different canvas and different camera.
  • Only one view is created as interactive: it’s canvas is set as engine.inputElement and it’s camera is controlled.
    Situation on screenshot:
  • You see Dude with spheres created at positions of his bones.
  • View at the top is created first.
  • View at the top (when created as interactive):
    → Always works for picking meshes for gizmos.
    → Breaks for selecting gizmos axes as soon as camera is moved*.
  • View at the bottom (when created as interactive - like on a attached screenshot):
    → Always works for selecting gizmos axes.
    → Breaks for picking meshes as soon as camera is moved*.

*You can still select mesh or axis, but by clicking in completely different place than the thing is rendered in on canvas you interact with.

My naive interpretation of the issue:
It seems as if raycast for picking mesh for gizmo is based on one camera, and raycast for picking gizmo elements (axes etc), is based on another camera.

What I tried:
1. Setting both cameras as controllable - this does fix the issue but misses point of one view being static.
2. Setting utility layer for gizmoManager and setting its renderedCamera to the controllable - no effect.
3. Initializing gizmoManager at different points in time relative to views creation (before after), or scene render (before first render, after first render, much after first render) - no effect.
4. Setting camera that is supposed to be controlled as active - no effect (well, apart from gizmo not working at all this time).
5. Setting camera that is supposed to be controlled as cameraToUseForPointers on scene - again, only breaking things completely.

Screenshot:
(These are not two screenshots stacked, it’s a single scene on 2 canvases.)

1 Like

I think you will need to provide a simple repro in the playground for us to be able to help.

There has been a change lately that seems related:

Have you tried with the very latest Babylon version? With 4.2?

1 Like

It was on alpha.25, now bumped to 31.
I prepared testing setup but results are quite boring - it just works :smile:
(https://playground.babylonjs.com/#T9VBPD#13 - top canvas is controlled and gizmo works great in all extent.)

In my project situation is in principle the same (2 canvases, 2 cameras, 2 views etc).
But I admit that overall setup is a bit more elaborate, I’m probably missing something.

If I’ll learn anything interesting when debugging I’ll post update.

3 Likes

After some investigation I arrived at: myUtilLayerScene.attachControl().

From practical standpoint it solves the problem.
Minor issue is that gizmo elements are not highlighted on hover.

However, looking at UtilityLayerRenderer it seems that calling attachControl() is a dirty workaround at best. (Or is it OK?)

I may add that I checked code of that playground I linked previously locally in my project and sadly could see that it fails aswell (i.e. act as if utility layer would not be set-up or listen on events on ‘wrong’ canvas). Even weirder - it would start working whenever Webpacks’ HMR would reload the app :see_no_evil:.

I looked at UtilityLayerRenderer code a bit but I understand it too little to suggest really anything with much confidence.
Just thinking out loud:

  • Is there any way for utility layer to end up doing picks from perspective of other camera than explicitly set renderCamera?
  • Can those picks somehow be ‘1 canvas off’?
  • What ifluence on scene.pointerX/Y of pointing device has initial canvas used for engine bootstrap? (In multi canvas it’s just ‘working’ canvas that is not a render target really, nor - as I understand - needs to correspond in any way geometrically with canvases for actual rendering.)

:thinking: :dizzy_face: :exploding_head: … I’m open to suggestions :grinning_face_with_smiling_eyes:

Let me ping our expert @Cedric (but please be patient as he is on vacations :))

1 Like

In case that seeing issue I describe in-the-flesh would help, I’ve prepared a simple repo with Vue.js application (similar to my project) that is using the exact code from the playground.

It is just a super simple demo app.
Only important files are HelloWorld.vue and gizmoScene.js.

To run it just: yarn then yarn serve, or alternatively npm i then npm run serve.

Is it not possible to repro in the playground instead ? as this is always way easier for the community to help.

Yes, the playground was linked before, but I’ll link it once again: https://playground.babylonjs.com/#T9VBPD#13

The problem is that the “exact same”* code behaves differently in the application.

*Well, not really the same, because Playground does not allow me to see under the hood - maybe there is the critical discrepancy? I’d like to learn if that’s so.

I’ve just tried to include raw utilityLayerRenderer.ts in my code to do some proper tinkering, and to my surprise, just using that local module instead of one imported from @babylonjs/core solved the issue completely.

It’s really fun to see gizmo finally working at 100% with 2 canvas locally :tada:, but on the other hand I think I’d prefer to debug and even fix the implementation than learn that module resolution in an issue :sob:

Are there any known rules of thumb for setup with webpack?
Or known no-nos that put robust module resolution and bundling of Babylon project at risk?

I have to admit I am fully confused it could work like this :slight_smile: I am not aware of any gotchas of the sort.

only thing is to not import * as BABYLON from '@babylonjs/core'

as this would conflict with some of the legacy think we were doing so you could instead:

import * as MYBABYLON from '@babylonjs/core' :slight_smile:

2 Likes

I was not aware of that, thank you! :slight_smile:

I now have made change to avoid clash with BABYLON, but sadly it was not the culprit in this case.

This issue is a good wake-up call for me. I expect (hope really) that project of mine to grow, and it’s better to spot build pipeline problems sooner rather than later.

So many variables come to mind as to what exactly may cause Webpack to get confused.

I intend to fix it. Probably starting with very simple hand-made setup.
I’ll keep updating this thread.

And thanks you all guys for being responsive!


In case anyone would have an inclination to ponder for a moment this issue - I’ll just echo one interesting fact I mentioned in one previous post - when Webpacks’ Hot Module Replacement kicks-in and does it hot-module-reloading magic refreshing app at runtime, everything starts working as it should :man_shrugging:

Did you try out outside of vue ? as it sounds more and more like a timing issue ?

I was short on time for fun things recently.

Just did manual setup from the ground and no more issues with UtilityLayerRenderer.

I’ll try add Vue and see how it will fare then.

1 Like

Vue added and all is working just fine.
It’s a bummer that the run-of-the-mill vue-cli setup (it’s like a create react app) with BJS added was having an issue.

For the posterity - if someone would have problems with Vue, some hard to track build issues and just need a working circa 2021 setup, I’ll paste my deps, webpack config and tsconfig.
If you’re not using Vue you’ll want to removue vue-loader and compilation options related to single file components.

#deps

"dependencies": {
    "@babylonjs/core": "^5.0.0-alpha.31",
    "@babylonjs/loaders": "^5.0.0-alpha.31",
    "@babylonjs/materials": "^5.0.0-alpha.31",
    "@babylonjs/serializers": "^5.0.0-alpha.31",
    "vue": "^3.0.0"
  },
  "devDependencies": {
    "@typescript-eslint/eslint-plugin": "^4.18.0",
    "@typescript-eslint/parser": "^4.18.0",
    "@vue/compiler-sfc": "^3.1.5",
    "@vue/eslint-config-typescript": "^7.0.0",
    "css-loader": "^6.2.0",
    "eslint": "^6.7.2",
    "eslint-plugin-vue": "^7.0.0",
    "html-webpack-plugin": "^5.3.2",
    "sass": "^1.26.5",
    "sass-loader": "^8.0.2",
    "source-map-loader": "^3.0.0",
    "style-loader": "^3.2.1",
    "ts-loader": "^9.2.4",
    "typescript": "~4.1.5",
    "vue-loader": "^16.4.0",
    "vue-style-loader": "^4.1.3",
    "webpack": "^5.47.1",
    "webpack-cli": "^4.7.2",
    "webpack-dev-server": "^3.11.2"
  }

#webpack.config.js

const path = require("path")
const fs = require("fs")
const HtmlWebpackPlugin = require("html-webpack-plugin")
const webpack = require('webpack');
const { VueLoaderPlugin } = require('vue-loader')
const appDirectory = fs.realpathSync(process.cwd())

const webAppEntryPoint = 'public/index.html'
const moduleResolutionEntryPoint = 'src/app.ts'
const contentDirectory = 'public'
const contentBase = path.resolve(appDirectory, contentDirectory)
const jsBundlePath = 'js/app.js'
const port = 9001
const https = false
const host = '0.0.0.0'
const publicPath = '/'

const srcAlias = '@'

module.exports = {
  entry: path.resolve(appDirectory, moduleResolutionEntryPoint), //path to the main .ts file
  output: {
    /* Name of emitted/held-in-memory js file */
    filename: jsBundlePath,
    /* Clean output dir before emit. Affects only production builds. */
    clean: true
  },
  devtool: 'source-map',
  resolve: {
    extensions: [".ts", ".js"],
    alias: {
      [srcAlias]: path.resolve(__dirname, 'src/')
    },
  },
  devServer: {
    host,
    port,
    disableHostCheck: false,
    /* Path to expose bundle from to the browser. (e.g. '/' -> http://localhost:8080/myBundle.js) */
    publicPath,
    /*
      Directory to serve static content from (bundle is in memory).
      It 'physical' path corresponding to actual directory where you hold your assets.
    */
    contentBase,
    hot: true,
    https
  },
  module: {
    rules: [
      {
        /* Only for Vue.js users */
        test: /\.vue$/,
        loader: 'vue-loader'
      },
      {
        test: /\.tsx?$/,
        loader: 'ts-loader',
        options: {
          /* Allows loading TS from SFCs */
          appendTsSuffixTo: [/\.vue$/],
        },
        exclude: /node_modules/,
      },
      {
        test: /\.s?css$/,
        use: [
          // Creates `style` nodes from JS strings
          // "style-loader",
          /* Allows loading styles from SFCs */
          'vue-style-loader',
          /* Translates CSS into CommonJS */
          "css-loader",
          /* Compiles Sass to CSS */
          "sass-loader"
        ],
        exclude: /node_modules/,
      },
      /* Source maps */
      {
        test: /\.tsx?$/,
        enforce: 'pre',
        loader: 'source-map-loader',
      },
      {
        test: /\.js/,
        resolve: {
          /*
            Temporary bug-fix.
            See: https://forum.babylonjs.com/t/modulenotfounderror-after-update-to-babylon-5-0-0-alpha/22638/2
          */
          fullySpecified: false,
        },
        enforce: 'pre',
        loader: 'source-map-loader',
      },
    ]
  },
  plugins: [
    new HtmlWebpackPlugin({
      inject: true,
      template: path.resolve(appDirectory, webAppEntryPoint),
    }),
    new VueLoaderPlugin(),
    new webpack.ProgressPlugin((percentage, message) => {
      console.log(`${(percentage * 100).toFixed()}% ${message}`);

      if (percentage*100 >= 100) {
        console.log(`------------------------------------`)
        console.log(`🟢 Project is running at: ${ 'http' + (https ? 's' : '') + '://' + host + ':' + port + '/'}`)
        console.log(`🟢 Webpack output is served from: ${publicPath}`)
        console.log(`🟢 Content not from webpack is served from: ${contentBase}`)
        console.log(`------------------------------------`)
      }
    })
  ],
  mode: "development",
}

#tsconfig.json

{
  "compilerOptions": {
    /* Visit https://aka.ms/tsconfig.json to read more about this file */

    "target": "es6",                          /* Specify ECMAScript target version: 'ES3' (default), 'ES5', 'ES2015', 'ES2016', 'ES2017', 'ES2018', 'ES2019', 'ES2020', or 'ESNEXT'. */
    "module": "commonjs",                     /* Specify module code generation: 'none', 'commonjs', 'amd', 'system', 'umd', 'es2015', 'es2020', or 'ESNext'. */
    "lib": [
      "DOM",
      "ES6"
    ],                             /* Specify library files to be included in the compilation. */
    "allowJs": false,                       /* Allow javascript files to be compiled. */
    "sourceMap": true,                     /* Generates corresponding '.map' file. */
    "rootDir": "src",                       /* Specify the root directory of input files. Use to control the output directory structure with --outDir. */
    "strict": true,                           /* Enable all strict type-checking options. */
    "paths": {
      "@/*": ["./src/*"]
    },                           /* A series of entries which re-map imports to lookup locations relative to the 'baseUrl'. */
    "esModuleInterop": true,                  /* Enables emit interoperability between CommonJS and ES Modules via creation of namespace objects for all imports. Implies 'allowSyntheticDefaultImports'. */
    "skipLibCheck": true,                     /* Skip type checking of declaration files. */
    "forceConsistentCasingInFileNames": true  /* Disallow inconsistently-cased references to the same file. */
  }
}

1 Like

thanks a ton for sharing it

2 Likes