setActionOnTouch callback in Class VirtualJoystick

Dear community,

I’m using version 7.2.1 of BabylonJS and trying to master virtual joystick control. I needed to implement a double-tap on the joystick, so I started reading the documentation and found a mention of the setActionOnTouch callback. However, it doesn’t seem to work for me.

Could you please advise on how to correctly capture events so that I can add various types of taps and gestures to the joystick’s regular movements?

I’m providing a link to a PlayGround for reference: link

If you have any experience or insights on this matter, please share.

cc @RaananW who should be able to advise (but be patient he is travelling right now :))

1 Like

Hey, and sorry for a late reply.

Is it possible that the playground is not complete? I don’t see any reference to setActionOnTouch.

In any way - setActionOnTouchis more of a way to define what happens when a single click is capture. Checking for doubletap is a bit different. However, since this is a virtual keyboard, you can simply use babylon’s pointer system to detect a double tap on the screen. If you know on what side you want it to work, you can calculate if the double tap was done in the place you expect it to be, and then trigger any action you want, using the observable’s callback mechanism.

Let me know if something doesn’t make sense :slight_smile:

Hello RaananW and thank you for reply.
Let’s take a systematic look at the VirtualJoystick element itself.
Does it violate OOP principles and encapsulation?

  • Did we give permissions to go beyond the scene boundaries?
  • Yes, we allow the joystick to go beyond the scene boundaries.
  • Did we allow it to go beyond the canvas boundaries?
  • No.

We did not authorize elements to be created outside of the canvas’s responsibility area. Perhaps the implementation was hindered by the task being difficult to accomplish within a single canvas, so the developer went on to create an additional canvas, going beyond the parent element into the body and started creating HTML elements there.
Let’s open the PlayGround. To even access our elements, we’ll have to remove the foreign element <canvas width="1235" height="963" touch-action="none" style="width: 100%; height: 100%; position: absolute; background-color: transparent; top: 0px; left: 0px; z-index: 5; touch-action: none;"></canvas>, which blocks access to the application. To be fair, it’s worth noting that setActionOnTouch did start working in this example: link to playground.babylonjs.com. So, we can consider this issue closed. With the Virtual Joystick enabled, all my touch screen events in my application are overshadowed by the joystick’s canvas. And only in fullscreen canvas mode do they start working, while the Joystick naturally remains outside the canvas boundaries in fullscreen mode, as it decided it could go beyond the canvas and take over my application by creating its elements outside the canvas. If a class hinders the application’s functionality and exhibits such behavior, it’s an alien class unrelated to BabylonJS.

Perhaps :face_with_raised_eyebrow: I’m being too harsh on this solution. Please forgive me for that. I don’t have a final opinion on this matter yet. But I really dislike such an implementation where I can’t use my application in fullscreen mode while keeping all its elements in view.

Hi!

Oh wow, that question took an interesting turn :slight_smile:

The VirtualJoystick class is at least 8 years old. And software is not like good wine - it doesn’t age well! I am not sure who the implementer of the class is, so I am not sure as to what was the train of thought behind it. However, everything can be fixed.
The reason I was pinged is that I know the babylon input system quite well, and can help, in general, on the matter. As such, I will also be happy to solving the issues you found there.
I agree that the element should not paint itself on top of anything other than the canvas. I am not sure what you mean about permission, but it is plain wrong to overlay a touch-blocking element on top of everything on the page. One thing i can say is that it used to work well in earlier versions, so something was broken along the way. I am not sure how this is related to OOP principals, but I guess this is different discussion to have :slight_smile:
The reason for using an external canvas is that the single canvas that you are talking about has 3D context. And the virtual joystick requires 2D context.

Anyhow, I will be happy to look into fixing those issues in the future. i’ll assign it to myself and will take it from here.

Hello RaananW, Obi-Wan Kenobi,

I wanted to share a quick report on an issue I encountered with the setActionOnTouch function while using VirtualJoystick. It seems the event wasn’t being attached to the correct joystick :joystick:, causing some confusion. :worried:

After investigating, I realized the problem stems from the argument used when creating the VirtualJoystick. When I used false for “leftJoystick,” it actually pointed to the right joystick, leading to unexpected behavior where the event triggers on the left side of the screen. Conversely, specifying true for the left joystick made it activate on the right side.

This seems to be a small oversight in the setActionOnTouch function’s implementation, where the joystick sides are swapped during creation.

Though I’m still learning to create Pull Requests for BabylonJS, I’m keen to help with testing and consider it my contribution to product quality.

Best wishes, Alex

1 Like

Thank you so much for sharing this. It’s very helpful to get real experience with the class, rather than my “theoretical” one :smile:
I’m traveling at the moment (and have been for the last week), so I will only have time to look into that after Thursday. I’ll check what happened along the way. We’ll get it working again. And I’ll make sure to send you a copy of the PR snapshot so you can check ensuring is working as expected.

VirtualJoystick still alive. I think it still awesome class. I want to share with you my add-on for it.

It’s my Touches.ts file.

import { VirtualJoystick } from 'babylonjs'

export interface SwipeDirection {
  up: boolean
  down: boolean
  right: boolean
  left: boolean
}

class Touches {

  lastStartTouchTime: number = 0
  lastEndTouchTime: number = 0
  lastStartHoldTime: number = 0
  lastEndHoldTime: number = 0
  lastStartTapTime: number = 0
  lastEndTapTime: number = 0
  lastEndSwipeTime: number = 0
  swipe: SwipeDirection = {
    up: false,
    down: false,
    right: false,
    left: false
  }
  tap: boolean = false
  touch: boolean = false
  hold: boolean = false
  holdCenter: boolean = false
  startTouch: number = 0
  endTouch: number = 0
  startHold: number = 0
  endHold: number = 0
  startTap: number = 0
  endTap: number = 0
  endSwipe: number = 0

  constructor(private stick: VirtualJoystick) {}

  detect(): [SwipeDirection, boolean, boolean, boolean] {
    const { pressed, deltaPosition } = this.stick
    const currentTime = new Date().getTime()
    this.startTouch = currentTime - this.lastStartTouchTime
    this.endTouch = currentTime - this.lastEndTouchTime
    this.startHold = currentTime - this.lastStartHoldTime
    this.endHold = currentTime - this.lastEndHoldTime
    this.startTap = currentTime - this.lastStartTapTime
    this.endTap = currentTime - this.lastEndTapTime
    this.endSwipe = currentTime - this.lastEndSwipeTime

    const thresholdY = 0.02
    const thresholdX = 0.002
    const deltaY = deltaPosition.y
    const deltaX = deltaPosition.x

    if (pressed) {
      this.lastStartTouchTime = currentTime
      if (!this.tap) {
        this.lastStartTapTime = currentTime;
      }

      if (this.endTouch < 250 && this.startTouch < 250) {
        this.touch = true

        //detect swipes
        if (
          Math.abs(deltaY) > thresholdY &&
          Math.abs(deltaX) < thresholdX
        ) {
          this.swipe.up = deltaY > 0;
          this.swipe.down = deltaY < 0;
        } else if (
          Math.abs(deltaX) > thresholdX &&
          Math.abs(deltaY) < thresholdY
        ) {
          this.swipe.right = deltaX > 0;
          this.swipe.left = deltaX < 0;
        }
        if (this.swipe.up || this.swipe.down || this.swipe.left || this.swipe.right) {
          this.lastEndSwipeTime = currentTime;
        }


      } else if (this.endTouch > 250 && this.startTouch < 250) {
        if (!this.hold) {
          this.lastStartHoldTime = currentTime
        }
        this.touch = false
        this.hold = true

        if (this.startHold > 500 && this.startHold < 700 && Math.abs(deltaY) <= thresholdY && Math.abs(deltaX) <= thresholdX) {
            // when first half second touch
            this.holdCenter = true
        }
      }

    } else {

      // short tap
      if (!this.tap && this.startTap < 150 && this.endSwipe > 150) {
        this.tap = !(this.startTap < 150 && this.hold) && this.endTap > 50;
        this.lastEndTapTime = currentTime;
      } else {
        this.tap = false;
      }

      this.reset()
      this.lastEndTouchTime = currentTime
    }

    return [this.swipe, this.tap, this.hold, this.holdCenter]
  }


  reset() {
    this.hold = false
    this.holdCenter = false
    this.touch = false
    this.swipe = { up: false, down: false, right: false, left: false }
    this.stick.deltaPosition.y = 0
    this.stick.deltaPosition.x = 0
  }

}
export default Touches

In constructor of my TouchInput.ts I do this:

    this.leftJoystick = new VirtualJoystick(true)
    this.rightJoystick = new VirtualJoystick(false)
    this.rightTouches = new Touches(this.rightJoystick)
    this.leftTouches = new Touches(this.leftJoystick)

And then I put in handle of joysticks:

    this.scene.registerBeforeRender(() => {
      this.handleLeftStick()
      this.handleRightStick()
    })

and so in handleRightStick() I just get all gestures:
const [swipe, tap, hold, holdCenter ] = this.rightTouches.detect()

And my main class with joystick become more cleaner without checks and detections of swipes or long touches. :smiley:
I understand that it would be proper to inherit a new class from VirtualJoystick and extend it by adding features like longPressed or SwipedUp, but I quickly put together this class, which I simply connect in the constructor. Who knows, someone might find it useful if they’re implementing swipes.

And also want to share function which filter stick data to make more smooth:

  private filterAxisDelta(delta: number, threshold: number = 0.0001): number {
    const sign = Math.sign(delta)
    delta = sign * delta * delta
    if (delta > threshold) {
      delta -= threshold
    } else if (delta < -threshold) {
      delta += threshold
    } else {
      delta = 0
    }
    return delta
  }

With some improvements, VirtualJoystick is actually a wonderful class. :slight_smile: I don’t plan to give it up for now. Thank you for adding it. :hugs:

1 Like

Want to submit a PR with fixes/extensions? Would be wonderful to be able to debug this as part of the core package (if you feel like contributing, of course)

I’m not sure yet if my code is worthy of inclusion in the main framework. I want to test it thoroughly first to ensure it doesn’t break any functionality, and then merge it with the main VirtualJoystick. I believe further testing on different devices is necessary to make sure I’ve truly created something worthwhile, rather than just a makeshift solution. But ultimately, yes, I would like to make a Pull Request to the main framework, because being listed as a contributor to BabylonJS will make it easier for me to find work and leverage it in job interviews.

1 Like