Optical Flow -> Fluid

Demos and projects
1

Hey guys, I have been working on migrating some computer vision processes to WGSL and decided to take a short detour to build a proof of concept.

The shader chain uses Sobel edge detection and Optical Flow to process camera input data and convert motion within the frame to flow vectors for the fluid. I have been working to optimize it and it runs, for the most part, at 60+fps on most mobile platforms. The fluid shader will detect desktop and add a few additional layers of processing, for shaping and esthetic, but I am pleased enough with the appearance of the mobile result. I added a camera switch and some sliders for tuning the fluid effect, and allowing for playback; something akin to an animated filter (which shows a 4:5 frame by default).

In order to tune the simulation I added some interfaces for the vision inputs (sobel and of), and you can tune them and see how they effect the end result (controllers store data between sessions).

I am hunting down some performance issues in iOS, which appear to be related to texture resource syncing in the fluid effect; refresh seems to handle it, but I only have an iPhone 13 to test on and would be interested to hear your experience. Please also bear with the load times (first load is about 4-5s) as I figure out some shader decryption work which will make that faster.

Let me know how it goes, and particularly if you find any bugs or issues.

Robert

You need WebGPU and it’s unlikely that mobile devices older than a few years old are going to run it

4 Likes
3

This looks great running live. Optical flow at 60fps on mobile is no joke for that pipeline. Most WGSL fluid demos I’ve seen pin a desktop GPU and call it a day. Curious if you hit any rolling-shutter weirdness with phone cameras during fast motion, or if the Sobel pass smooths it out enough that it doesn’t show.

4

Hey, and thanks. I haven’t seen any skewing artefacts or issues with shutter, save for iOS struggling with mixed frame targets (32fp → 8uint). The fluid is probably the most expensive piece in the pipeline, and the sobel->of are only capped by the scheduler running them at 60fps. I am working on adding FAST Harris (feature detection) between the sobel and OF layers, as my goal is to reproject and resolve a camera transform (with enough resources left to run a 3d scene).

I’ll post an updated demo in a couple weeks, and thanks again.

r

5

After some fun in the Apple store I was able to confirm that the issue I found on iPhone 13 (slowness from mixed RTT formats) doesn’t exist on more recent devices (15+). Please let me know if you happen to have an iPhone 14 (wasn’t available to test against).

The qr code at the bottom of the ux will direct other users directly to the page for easy sharing.

r