The goal would be to make the container fully ‘swippable’ and not only on the scroll bar. To help you understand what I have in mind, you can look at the application : Google Play or AppStore where users can drag content horizontally.
Actually, you can just set the bar size to zero. Next you can move on desktop with the mouse wheel.
However, the issue on mobile remains as you said:
You have to first click on the bar to unlock the gesture on the scrollviewer.
Yes, this is I believe the go-around we found for both mobile and desktop. Detect whether you select the root or the sv container and use pointer events.
Edit: Here I just quickly copied the pick logic from this other case study. Commented parts are because the scene was using a 2 cameras approach. But basically the idea is that on pick (either scene,root or sv) you enact or disable the pointer events. Hope this helps,