VirtualJoystick with swipe, tap, hold detections

onekit · May 5, 2024, 8:30pm

I created class which extends BABYLON.VirtualJoystick. I share it for you:
https://github.com/onekit/babylonjs-touchstick
Give a star if you like this idea.

BabylonJS TouchStick

This TypeScript class extends the functionality of the standard VirtualJoystick in BabylonJS. It provides smooth motion delta with movement filtering, and gesture information such as swipe, hold, hold center, and tap. Additionally, you can directly obtain Vector3 from the joystick for controlling the character.

Install

NPM

npm i babylonjs-touchstick

Yarn

yarn add babylonjs-touchstick

Parameters

Tap

tap: Represents a quick single touch.
doubleTap: Represents a quick double tap.

Swipe

swipe.up: Denotes quick movement upwards on the joystick.
swipe.down: Denotes quick movement downwards on the joystick.
swipe.left: Denotes quick movement to the left on the joystick.
swipe.right: Denotes quick movement to the right on the joystick.

Hold

hold: Indicates holding the joystick.
holdCenter: Indicates holding the joystick immediately after touching it in the center.

Direction

direction: Represents a BABYLON.Vector3 type for character direction. The direction.length() can be used as delta for movement.

Delta Smoothing

Original deltaPosition, but with smoothing.

deltaPositionSmoothed: Represents smoothed movements of delta using cube easing. You can use deltaPositionSmoothed.x and deltaPositionSmoothed.y instead of deltaPosition.x and deltaPosition.y.

Usage

TypeScript example:

import TouchStick from 'babylonjs-touchstick'
import { AbstractMesh } from 'babylonjs'

export class TouchInput {

    private stickLeft: TouchStick = new TouchStick(true);
    private stickRight: TouchStick = new TouchStick(false);

    constructor() {
        this.stickLeft.setDirectionMaxLength(3) // cap for maximum distance of BABYLON.Vector3 length
    }

    private handleEvents() {
        const {swipe, tap, doubleTap, hold, holdCenter} = this.stickRight;

        if (swipe.down) {
            console.log('Swiped down right stick')
        }

        if (doubleTap) {
            console.log('Double tap')
        }

        if (holdCenter) {
            console.log('Enter menu')
        }

        if (this.stickLeft.swipe.up || this.stickRight.swipe.up) {
            console.log('Swiped up both sticks')
        }

    }
    
    handleMovement(mesh: AbstractMesh) {
        mesh.position
            .add(this.stickLeft.direction)
        
        mesh.position
            .scale(this.stickLeft.direction.length())
    }

}

Read official documentation of original BABYLON.VirtualJoystick for more information: Babylon.js docs

Upd May 21, 2024: Fine-tuned swipes detection and restructured class TouchStick.ts:

import { Vector3, VirtualJoystick } from 'babylonjs'

class TouchStick extends VirtualJoystick {
    direction: Vector3 = Vector3.Zero()
    private directionMaxLength: number = 2.8
    private directionSensitivity: number = 12000
    deltaPositionSmoothed: {
        x: number
        y: number
    } = { x: 0, y: 0 }
    swipe: {
        up: boolean
        down: boolean
        right: boolean
        left: boolean
    } = {
        up: false,
        down: false,
        right: false,
        left: false,
    }
    tap: boolean = false
    doubleTap: boolean = false
    hold: boolean = false
    holdCenter: boolean = false
    threshold: number = 0
    private lastStartTouchTime: number = 0
    private lastEndTouchTime: number = 0
    private lastStartHoldTime: number = 0
    private lastStartHoldCenterTime: number = 0
    private lastStartTapTime: number = 0
    private lastEndTapTime: number = 0
    private lastEndSwipeTime: number = 0
    private lastEndHoldTime: number = 0
    private startTouch: number = 0
    private endTouch: number = 0
    private startHold: number = 0
    private startHoldCenter: number = 0
    private startTap: number = 0
    private endTap: number = 0
    private endSwipe: number = 0
    private endHold: number = 0

    getThreshold(): number {
        return this.threshold
    }

    setThreshold(threshold: number) {
        this.threshold = threshold
    }

    getDirectionSensitivity(): number {
        return this.directionSensitivity
    }

    setDirectionSensitivity(directionSensitivity: number) {
        this.directionSensitivity = directionSensitivity
    }

    getDirectionMaxLength() {
        return this.directionMaxLength
    }

    setDirectionMaxLength(directionMaxLength: number) {
        this.directionMaxLength = directionMaxLength
    }

    constructor(isLeftStick: boolean) {
        super(isLeftStick)
        this.setJoystickSensibility(1)
        this.setupListener()
    }

    private setupListener = () => {
        this.detect()
        this.deltaPositionSmoothed = this.smoothDeltaPosition()
        this.direction = this.getDirection(
            this.deltaPosition.x,
            this.deltaPosition.y,
        )
        requestAnimationFrame(this.setupListener)
    }

    detect() {
        const { pressed, deltaPosition } = this
        const currentTime = new Date().getTime()
        this.startTouch = currentTime - this.lastStartTouchTime
        this.endTouch = currentTime - this.lastEndTouchTime
        this.startHold = currentTime - this.lastStartHoldTime
        this.endHold = currentTime - this.lastEndHoldTime
        this.startHoldCenter = currentTime - this.lastStartHoldCenterTime
        this.startTap = currentTime - this.lastStartTapTime
        this.endTap = currentTime - this.lastEndTapTime
        this.endSwipe = currentTime - this.lastEndSwipeTime

        if (pressed) {
            this.lastStartTouchTime = currentTime
            this.detectSwipes(deltaPosition)
            this.detectHold(currentTime)
            this.detectHoldCenter(currentTime, deltaPosition)
            if (!this.tap) {
                this.lastStartTapTime = currentTime
            }
        } else {
            this.detectTap(currentTime)
            this.lastEndTouchTime = currentTime
            this.reset()
        }
    }

    private detectSwipes(deltaPosition: { x: number, y: number }) {
        const thresholdY = 0.1
        const thresholdX = 0.1
        const deltaY = deltaPosition.y
        const deltaX = deltaPosition.x

        if (this.endTouch < 400 && this.startTouch < 400) {
            if (Math.abs(deltaY) > thresholdY && Math.abs(deltaX) < thresholdX) {
                this.swipe.up = deltaY > 0
                this.swipe.down = deltaY < 0
            } else if (Math.abs(deltaX) > thresholdX && Math.abs(deltaY) < thresholdY) {
                this.swipe.right = deltaX > 0
                this.swipe.left = deltaX < 0
            }
            if (this.swipe.up || this.swipe.down || this.swipe.left || this.swipe.right) {
                this.lastEndSwipeTime = new Date().getTime()
            }
        }
    }

    private detectHold(currentTime: number) {
        if (this.endTouch > 600 && this.startTouch < 600) {
            if (!this.hold) {
                this.lastStartHoldTime = currentTime
            }
            this.hold = true
        } else {
            this.lastEndHoldTime = currentTime
        }
    }

    private detectHoldCenter(currentTime: number, deltaPosition: { x: number, y: number }) {
        const thresholdY = 0.01
        const thresholdX = 0.01
        const deltaY = deltaPosition.y
        const deltaX = deltaPosition.x

        if (this.startHold < 900 && this.hold && !this.holdCenter && Math.abs(deltaY) <= thresholdY && Math.abs(deltaX) <= thresholdX) {
            this.lastStartHoldCenterTime = currentTime
            this.holdCenter = true
        } else if (this.startHoldCenter >= 900) {
            this.holdCenter = false
        }
    }

    private detectTap(currentTime: number) {
        if (!this.doubleTap && !this.tap && !this.hold && this.startTap < 50 && this.endSwipe > 350) {
            this.tap = !(this.startTap < 50 && this.hold) && this.endTap > 150 && (!this.hold && this.endHold < 250)
            if (this.tap) {
                if (this.endTap < 450) {
                    this.doubleTap = true
                    this.tap = false
                }
                this.lastEndTapTime = currentTime
            }
        } else {
            this.tap = false
            this.doubleTap = false
        }
    }

    private smoothDeltaPosition() {
        return {
            x: this.filterAxisDelta(this.deltaPosition.x, this.threshold) * 4500,
            y: this.filterAxisDelta(this.deltaPosition.y, this.threshold) * 4500,
        }
    }

    private getDirection(deltaX: number, deltaY: number): Vector3 {
        const deltaPosition = { x: deltaX, y: deltaY }
        const joystickVector = new Vector3(deltaPosition.x, 0, deltaPosition.y)
        if (joystickVector.length() === 0) {
            return Vector3.Zero()
        }
        const scaleFactor = 3
        const scaledLength =
            Math.pow(joystickVector.length(), scaleFactor) * this.directionSensitivity
        const cappedLength = Math.min(scaledLength, this.directionMaxLength)
        return joystickVector.normalize().scale(cappedLength)
    }

    private filterAxisDelta(delta: number, threshold: number = 0.00002): number {
        delta = delta * delta * delta
        if (delta > threshold) {
            delta -= threshold
        } else if (delta < -threshold) {
            delta += threshold
        } else {
            delta = 0
        }
        return delta
    }

    private reset() {
        this.direction = Vector3.Zero()
        this.hold = false
        this.holdCenter = false
        this.swipe = { up: false, down: false, right: false, left: false }
        this.deltaPosition.y = 0
        this.deltaPosition.x = 0
    }
}

export default TouchStick

rafaelj · May 5, 2024, 10:49pm

Hi,
First of all, thanks a lot for sharing your virtual control code. I really appreciate it.

I’ve downloaded the .ts file and integrated it into my game. However, during my initial test, I encountered a peculiar issue. The virtual control seems to only function properly on the left half of the screen. I’m wondering if you have any insights or some parameters to change into what might be causing this behavior? No erros in cosole.log.

Once again, thanks for your contribution!

onekit · May 6, 2024, 8:26am

Yes. Cause it in boundaries of stick (one side of screen). It’s correct if you use left stick, swipe need to work on left stick. On right stick you can use another object:

private stickLeft: TouchStick = new TouchStick(true);

private stickRight: TouchStick = new TouchStick(false);

In VirtualJoystick the parameter means “leftJoystick”.

However I admit need more tunes for sensitivity and detections.

To detect both sides for swipe we can use this check:

if (this.rightStick.swipe.up || this.leftStick.swipe.up) {
  console.log('swiped up both sides')
}

rafaelj · May 6, 2024, 10:31am

Thanks, i see now: constructor(isLeftStick: boolean) {

All right now, or left

onekit · May 6, 2024, 6:27pm

Yes. And today I added doubleTap feature. Upgraded and decide to make it easy to use. Looks funny 10kb script as separate packet in package.json. Now I add it as dependency in my project file: "babylonjs-touchstick": "^0.0.3" and will support it. Yeah. I am almost real contributor of BabylonJS.

babylonjs-touchstick v.0.0.3

This TypeScript class extends the functionality of the standard VirtualJoystick in BabylonJS. It provides smooth motion delta with movement filtering, and gesture information such as swipe, hold, hold center, and tap. Additionally, you can directly obtain Vector3 from the joystick for controlling the character.

Install

NPM

npm i babylonjs-touchstick

Yarn

yarn add babylonjs-touchstick

Parameters

Tap

tap: Represents a quick single touch.
doubleTap: Represents a quick double tap.

Swipe

swipe.up: Denotes quick movement upwards on the joystick.
swipe.down: Denotes quick movement downwards on the joystick.
swipe.left: Denotes quick movement to the left on the joystick.
swipe.right: Denotes quick movement to the right on the joystick.

Hold

hold: Indicates holding the joystick.
holdCenter: Indicates holding the joystick immediately after touching it in the center.

Direction

direction: Represents a BABYLON.Vector3 type for character direction.

Delta Smoothing

Original deltaPosition, but with smoothing.

deltaPositionSmoothed: Represents smoothed movements of delta using cube easing. You can use deltaPositionSmoothed.x and deltaPositionSmoothed.y instead of deltaPosition.x and deltaPosition.y.

Usage

TypeScript example:

import TouchStick from 'babylonjs-touchstick'

export class TouchInput {

    private stickLeft: TouchStick = new TouchStick(true);
    private stickRight: TouchStick = new TouchStick(false);
    
    private handleEvents() {
        const { swipe, tap, doubleTap, hold, holdCenter } = this.stickRight;
        
        if (swipe.down) {
            console.log('Swiped down right stick')
        }
        
        if (doubleTap) {
            console.log('Double tap')
        }

        if (holdCenter) {
            console.log('Enter menu')
        }
        
        if (this.stickLeft.swipe.up || this.stickRight.swipe.up) {
            console.log('Swiped up both sticks')
        }
        
    }

}

Read official documentation of original BABYLON.VirtualJoystick for more information: Babylon.js docs

rafaelj · May 6, 2024, 9:02pm

Have you ever thought about adding a parameter to give the option of showing or not showing the visual part of the control, in this case the circles?

onekit · May 14, 2024, 12:44pm

This feature already exists in base class of VirtualJoystick. I use transparent color for it.

stick = new TouchStick(true)
stick.setJoystickColor('rgba(0,0,0,0)')

or put your image instead of original:

stick.setContainerImage('/img/something.svg')

Looks like hack, I know.

BTW: changing image of puck

stick.setPuckImage('/img/puck.svg')

sounds interesting, however I cover it with my finger.

rafaelj · May 14, 2024, 1:20pm

Great!

Thanks!

rafaelj · May 23, 2024, 12:46am

Is it possible to use the TouchJoystick along with GUI buttons? According to the documentation, to use the touch joystick together with GUI buttons, we need to use a CustomJoystick type.

onekit · May 23, 2024, 6:24am

Yesterday I experimented with adding a feature that, like a curtain, extends and retracts virtual joysticks from the bottom of the screen.

const stick = new TouchStick(true)
stick.enableSwipeSwitcher(scene)

But I couldn’t think of a way for the sticks to work simultaneously with the main canvas.

const stick = new TouchStick(true)
stick.enableCanvasManager()
stick.canvasManager.show()
stick.canvasManager.hide()

For now, I’m trying to hide and show the canvas, although I understand that this solution requires a radically different approach and the Virtual Joystick does not have the right to create a separate canvas and must work within the boundaries of our application canvas.

My experiments here:

github.com

onekit/babylonjs-touchstick/blob/master/CanvasManager.ts

import { Nullable, Scene } from 'babylonjs'

export default class CanvasManager {
  private stickCanvas: HTMLCanvasElement | null = this.getCanvasWithoutId()
  private appCanvas: Nullable<HTMLCanvasElement> | undefined
  private swipe: {
    up: boolean
    down: boolean
    right: boolean
    left: boolean
  } = {
    up: false,
    down: false,
    right: false,
    left: false,
  }
  private swipeScreenPartApp = 0.85
  private swipeScreenPartStick = 0.9
  private threshold = 40 // Minimal distance for swipe detect
  private visible: boolean = true

This file has been truncated. show original

rafaelj · May 23, 2024, 5:47pm

I appreciate your efforts in finding a solution to the issue with the module not working with Babylon.js GUI, due to its extension of the VirtualJoystick class. I’m hopeful that you or others will manage to resolve it soon. Currently, I’m working on a project where, due to this problem, I’m opting for a Custom Joystick which would be more suitable. Looking forward to any updates or progress on this module and it compatibility with GUI.

Topic		Replies	Views
Babylonjs virtual joystick and scene actions Questions gui , camera , joystick	5	145	May 24, 2024
setActionOnTouch callback in Class VirtualJoystick Questions documentation , joystick	10	108	May 6, 2024
Does VirtualJoysticksCamera (& TouchCamera/UniversalCamera) still require hand.js? Questions	3	107	January 22, 2024
Tapping through virtual joystick canvas Questions	24	780	February 5, 2024
VirtualJoysticksCamera with onBlurObservable or onPointerClickObservable Questions gui , camera	1	336	August 25, 2022

VirtualJoystick with swipe, tap, hold detections

BabylonJS TouchStick

Install

NPM

Yarn

Parameters

Tap

Swipe

Hold

Direction

Delta Smoothing

Usage

TypeScript example:

babylonjs-touchstick v.0.0.3

Install

NPM

Yarn

Parameters

Tap

Swipe

Hold

Direction

Delta Smoothing

Usage

TypeScript example:

Related topics