I have some demo code that I think solves this and solves the other problem: Scrolling scrollViewer seems to not work on Mobile?
I wanted to point out that I did end up using the onPointerEnterObservable / Out as suggested by @PolygonalSun 
So thanks for everyoneâs help. Particularly @mawa ! 