Gaze controlled app


Do you guys have experience with gaze controlled apps? I already tried WebGazer and some smaller TensorFlow based solutions. The most reliable is WebGazer but performance is so poor that it eats almost all the resources on an RTX3080Ti. FPS is around 80-90 (and it’s running only the detection stuff) but it would be a joke to require such powerful HW to use this feature in the game. The CPU based calculations are useless. I am getting around 30 FPS and even less on a 5gig i9. (maybe multiple webworkers could help here)

I tried to leave out the eye movement detection and used only face detection. It’s millon times faster but this way the player needs to rotate his head and it’s not that cool and accurate.

I am thinking of utilizing custom color detection which works pretty fast and track the eye’s white and dark areas but I’m not sure it will work.

It’s so cool to aim a Vulcan cannon at the enemies and shoot missiles just by double blinking :sunglasses: so it would be quite painful to leave the feature out of the game so really any input is appreciated!

I did some research on other frameworks but don’t have enough spare time to try them all so guys if you can save me some time by shedding some light on this topic and have a working shareable solution I will be really happy and thankful!

Thanks a lot!

mediapipe is probably what you’re looking for

.GitHub - YunYouJun/vtuber: 👻 Make a vtuber from one. 从一开始的 Vtuber
.GitHub - yeemachine/kalidoface-3d: Face and Body Tracking for VRM 3D models on the web.
.GitHub - yeemachine/kalidokit: Blendshape and kinematics calculator for Mediapipe/Tensorflow.js Face, Eyes, Pose, and Finger tracking models.
.react-electron-facemesh/drawCanvas.ts at main · da-okazaki/react-electron-facemesh · GitHub


Thank you!

np, there is also face-api , here is a vue electron app using it in a worker

.GitHub - liz2020/Virtual-Live: Virtual-Live is a desktop program that aims to enable anyone to use a webcam to control live2d characters. This program is developed for Mac, but could be compatible with Windows and Linux after some modifications.

1 Like

:heart_eyes_cat: and thanks once again!

Ok, let me say, I’VE JUST DROPPED MY JAW after trying out Mediapipe! Thanks a lot! This was another trap, but a boobie one :joy::joy::joy: (I am sure only you’ll get the joke :rofl:). Not just the tracking stuff itself but the anime character is incredible as well! :heart_eyes_cat:

However the middle finger gesture is not working lol!

1 Like

I just don’t get why didn’t Google list this one for me when I was searching for available solutions :roll_eyes:

I couldn’t get Virtual Live working (some files seems to be missing), but I am so amazed with Kalidokit that I can’t wait to use it in my project!

1 Like

I’m using double blinking because I ran into the same issues as this dude and I am using it only for the homing missiles. You can launch them two at a time and they get reloaded only after they are destroyed. The main weapon is fired by the mouse button.
Thank you for the link! How cool is to realize that other people on the other side of the planet have the same iideas😜

Some of his videos are so funny

1 Like