IWebBrowser2 (.NET WebBrowser Control) does not respond to mouse events

Hello guys,

I have tried to make Babylon.JS run in WebBrowser control of .NET (32 bit application)

I have made a local htm file with content below, saved into a file Babylon.htm

To make WebBrowser work with local files, I have added then registry key for my demo application

Key:
HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\Internet Explorer\Main\FeatureControl\FEATURE_BROWSER_EMULATION
Value: DWORD WindowsApp1.exe 0x2af9

The WebBrowser control on a form of WindowsApp1 has got the url

file:///C:\Babylon\Babylon.htm

After running the app, the content of the page is displayed well, but no mouse interaction works. What can be done?

<!DOCTYPE html>
<html>
<head>
    <meta http-equiv="Content-Type" content="text/html" charset="utf-8"/>
    <title>Babylon - Getting Started</title>
    <!--- Link to the last version of BabylonJS --->
    <script src="https://cdn.babylonjs.com/babylon.js"></script>
    <style>
        html, body {
            overflow: hidden;
            width   : 100%;
            height  : 100%;
            margin  : 0;
            padding : 0;
        }

        #renderCanvas {
            width   : 100%;
            height  : 100%;
            touch-action: none;
        }
    </style>
</head>
<body>
    <canvas id="renderCanvas"></canvas>
    <script>
        window.addEventListener('DOMContentLoaded', function(){
            // get the canvas DOM element
            var canvas = document.getElementById('renderCanvas');

            // load the 3D engine
            var engine = new BABYLON.Engine(canvas, true);

            // createScene function that creates and return the scene
            var createScene = function(){
                // create a basic BJS Scene object
                var scene = new BABYLON.Scene(engine);

                // create a FreeCamera, and set its position to (x:0, y:5, z:-10)
                var camera = new BABYLON.FreeCamera('camera1', new BABYLON.Vector3(0, 5,-10), scene);

                // target the camera to scene origin
                camera.setTarget(BABYLON.Vector3.Zero());

                // attach the camera to the canvas
                camera.attachControl(canvas, false);

                // create a basic light, aiming 0,1,0 - meaning, to the sky
                var light = new BABYLON.HemisphericLight('light1', new BABYLON.Vector3(0,1,0), scene);

                // create a built-in "sphere" shape; its constructor takes 6 params: name, segment, diameter, scene, updatable, sideOrientation 
                var sphere = BABYLON.Mesh.CreateSphere('sphere1', 16, 2, scene);

                // move the sphere upward 1/2 of its height
                sphere.position.y = 1;

                // create a built-in "ground" shape;
                var ground = BABYLON.Mesh.CreateGround('ground1', 6, 6, 2, scene);

                // return the created scene
                return scene;
            }

            // call the createScene function
            var scene = createScene();

            // run the render loop
            engine.runRenderLoop(function(){
                scene.render();
            });

            // the canvas/window resize event handler
            window.addEventListener('resize', function(){
                engine.resize();
            });
        });
    </script>
</body>
</html>

Hello and welcome!

Did you make sure to add a reference to jQuery PEP?

Sorry, now the HTML is ready to watch.
What is to be done, please?

When I run the file directly in chrome everything is working…

<!--- Link to the last version of BabylonJS --->
<script src="https://code.jquery.com/pep/0.4.2/pep.min.js"></script>
<script src="https://cdn.babylonjs.com/babylon.js"></script>

as @Deltakosh mentioned, try that :slight_smile:

I have tried: nope - not working

Might be a problem, AddEventListener is connected to window, not document?

Tried with document at resize, not working

then i would assume it’s a “IWebBrowser2” related issue :thinking:
sorry, i haven’t got the slightest clue what that is for or how it works.
Maybe someone else does :slight_smile:

Have you tried googling mouse/pointer events in relation to this “IWebBrowser2” ?
Maybe they are different from regular browser’s events

OMG, thought you know it… :slight_smile:
No other way to display it on .NET forms project!?

The web browser is using IE so does it works in IE?

Yes it does.

I have investigated a bit, and found code incompatible with IE.

First of all, it seems

line with

if (n.isPointerLock && !n.isInVRExclusivePointerMode) {

is never true here. It never jumps into. When I ignore it with ||1 there is usage of movementX and movementY.

This is incompatible with IE and also Safari

But I’m not familiar with your code.

I always get r==0 !!

var a = o.clientX - i.previousPosition.x, c = o.clientY - i.previousPosition.y;
i.camera.getScene().useRightHandedSystem && (a *= -1), i.camera.parent && i.camera.parent._getWorldMatrixDeterminant() < 0 && (a *= -1), i._allowCameraRotation && (i.camera.cameraRotation.y += a / i.angularSensibility, i.camera.cameraRotation.x += c / i.angularSensibility), i.onPointerMovedObservable.notifyObservers({ offsetX: a, offsetY: c }), i.previousPosition = { x: o.clientX, y: o.clientY }, t || o.preventDefault();
}
}
}), this._onMouseMove = function (e) {
if ((n.isPointerLock && !n.isInVRExclusivePointerMode)||1) {
var r = e.movementX || e.mozMovementX || e.webkitMovementX || e.msMovementX || 0;
alert("MouseMove " + r);

                    i.camera.getScene().useRightHandedSystem && (r *= -1), i.camera.parent && i.camera.parent._getWorldMatrixDeterminant() < 0 && (r *= -1), i.camera.cameraRotation.y += r / i.angularSensibility;
                    var o = e.movementY || e.mozMovementY || e.webkitMovementY || e.msMovementY || 0;
                    i.camera.cameraRotation.x += o / i.angularSensibility, i.previousPosition = null, t || e.preventDefault();
                }
            }, this._observer = this.camera.getScene().onPointerObservable.add(this._pointerInput, Pe.a.POINTERDOWN | Pe.a.POINTERUP | Pe.a.POINTERMOVE), e.addEventListener("mousemove", this._onMouseMove, !1), e.addEventListener("contextmenu", this.onContextMenu.bind(this), !1);
        }, e.prototype.onContextMenu = function (e) {
            e.preventDefault();

Yes this is not the problem as this code is ignored on IE
So if your code works in IE, it should work in your webview as it is the same rendering engine

You tested with exactly the same page and inputs work in IE right?

Yes of course.
When I connect the MSHTML.HTMLDocument with the web browser control, I get the document Mouse events bubbled through.

I have copied the script of your Babylon locally and I’m editing it with Visual Studio. Because of no debugging possibilities of the WebBrowser control I have to make alerts in code to get some results.

It is using the IE on my machine IE11.885
I have found some notes that FEATURE_BROWSER_EMULATION with 0x2af9 will use edge, but that is unfortunately not true. WebBrowser control is using IE. I did actually not find the switch to change it.

If it’s possible for you to use Visual Studio it’s really easy to make a demo project to test that stuff…

Running the file directly in IE everything is working!
So I believe you are using anywhere anything to determine IE, but WebBrowser control returns something different.

So there are really differences. For example IE11 standalone echos navigator.userAgent

Standalone:
Mozilla/5.0 (Windows NT 10.0; WOW64; Trident 7.0; Touch; .NET4.0C; .NET4.0E; Tablet PC 2.0; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729; rv:11.0) like Gecko

WebBrowser
Mozilla/5.0 (Windows NT 6.2; WOW64; Trident/7.0; rv:11.0) like Gecko

After googling I found WebBrowser control is not supporting pointer events. It has a legacy fallback here.

Mouse movements in WebBrowser control are available with following code fragment:

    var ut = function () {
        function e(e) {
            void 0 === e && (e = !0), this.touchEnabled = e, this.buttons = [0, 1, 2], this.angularSensibility = 2e3, this.previousPosition = null, this.onPointerMovedObservable = new r.c, this._allowCameraRotation = !0;
        }
        return e.prototype.attachControl = function (e, t) {
            var i = this, n = this.camera.getEngine();
            this._pointerInput || (this._pointerInput = function (r) {
                var o = r.event;
                if (!n.isInVRExclusivePointerMode && (i.touchEnabled || "touch" !== o.pointerType) && (r.type === Pe.a.POINTERMOVE || -1 !== i.buttons.indexOf(o.button))) {
                    var s = o.srcElement || o.target;
                    if (r.type === Pe.a.POINTERDOWN && s) {
                        try {
                            s.setPointerCapture(o.pointerId);
                        }
                        catch (e) {
                        }
                        i.previousPosition = { x: o.clientX, y: o.clientY }, t || (o.preventDefault(), e.focus());
                    }
                    else if (r.type === Pe.a.POINTERUP && s) {
                        try {
                            s.releasePointerCapture(o.pointerId);
                        }
                        catch (e) {
                        }
                        i.previousPosition = null, t || o.preventDefault();
                    }
                    else if (r.type === Pe.a.POINTERMOVE) {
                        if (!i.previousPosition || n.isPointerLock)
                            return;
                        var a = o.clientX - i.previousPosition.x, c = o.clientY - i.previousPosition.y;
                        i.camera.getScene().useRightHandedSystem && (a *= -1), i.camera.parent && i.camera.parent._getWorldMatrixDeterminant() < 0 && (a *= -1), i._allowCameraRotation && (i.camera.cameraRotation.y += a / i.angularSensibility, i.camera.cameraRotation.x += c / i.angularSensibility), i.onPointerMovedObservable.notifyObservers({ offsetX: a, offsetY: c }), i.previousPosition = { x: o.clientX, y: o.clientY }, t || o.preventDefault();
                    }
                }
            }), this._onMouseMove = function (e) {
                if (e) {
                    if (e.buttons==1) {
                        if (i.previousPosition) {
                            var r = e.clientX - i.previousPosition.x || 0;
                            var o = e.clientY - i.previousPosition.y || 0;
                        } else {
                            var r = 0;
                            var o = 0;
                        }
                        i.camera.getScene().useRightHandedSystem && (r *= -1), i.camera.parent && i.camera.parent._getWorldMatrixDeterminant() < 0 && (r *= -1), i.camera.cameraRotation.y += r / i.angularSensibility;
                        i.camera.cameraRotation.x += o / i.angularSensibility, t || e.preventDefault();
                    }
                    i.previousPosition = { x: e.clientX, y: e.clientY };
                }
            }, this._observer = this.camera.getScene().onPointerObservable.add(this._pointerInput, Pe.a.POINTERDOWN | Pe.a.POINTERUP | Pe.a.POINTERMOVE), e.addEventListener("mousemove", this._onMouseMove, !1), e.addEventListener("contextmenu", this.onContextMenu.bind(this), !1);
        }, e.prototype.onContextMenu = function (e) {
            e.preventDefault();
        }, e.prototype.detachControl = function (e) {
            this._observer && e && (this.camera.getScene().onPointerObservable.remove(this._observer), this._onMouseMove && e.removeEventListener("mousemove", this._onMouseMove), this.onContextMenu && e.removeEventListener("contextmenu", this.onContextMenu), this.onPointerMovedObservable && this.onPointerMovedObservable.clear(), this._observer = null, this._onMouseMove = null, this.previousPosition = null);
        }, e.prototype.getClassName = function () {
            return "FreeCameraMouseInput";
        }, e.prototype.getSimpleName = function () {
            return "mouse";
        }, l.c([Object(L.c)()], e.prototype, "buttons", void 0), l.c([Object(L.c)()], e.prototype, "angularSensibility", void 0), e;
    }();

I have modified

            }), this._onMouseMove = function (e) {
                if (e) {
                    if (e.buttons==1) {
                        if (i.previousPosition) {
                            var r = e.clientX - i.previousPosition.x || 0;
                            var o = e.clientY - i.previousPosition.y || 0;
                        } else {
                            var r = 0;
                            var o = 0;
                        }
                        i.camera.getScene().useRightHandedSystem && (r *= -1), i.camera.parent && i.camera.parent._getWorldMatrixDeterminant() < 0 && (r *= -1), i.camera.cameraRotation.y += r / i.angularSensibility;
                        i.camera.cameraRotation.x += o / i.angularSensibility, t || e.preventDefault();
                    }
                    i.previousPosition = { x: e.clientX, y: e.clientY };
                }
            }

Of course this is not a fast solution and is only a sample for making it work with WebBrowser control. Babylon is using obviously _onMouseMove, when pointer events are not working. But here your code uses incompatible properties. So I’m calculating the delta movement of clientX and clientY positions.

i don’t know the control well enough to help more but definitely something is wrong in the way to support inputs…UNLESS…is the input capture by Winforms by any chance and not sent to the webbrowser??