OpenCV Integration

Questions
1

I’ve read a few forum topics somewhat related to this including Webdevelopment for WebVR and WebAr - #8 by player11 and this article https://babylonjs.medium.com/babylon-ar-7823ab4a80c1 and this post Marker Tracking and Webpiling - #5 by syntheticmagus

I think I am missing something when I’m reading the items.

I started with a giant description of my project and then thought it better I get to the point.

I have read the BabylonAR code and I am looking for the better way to translate the OpenCV coordinates to BabylonJS. Are the BabylonJS scene coordinates the same as WebGL coordinates?

I have OpenCV detecting objects and faces and sending those coordinates to BabylonJS that then adds a new HUD/GUI object (or removes it if the item disappears), but I’m manually messing with the coordinates in BabylonJS so the HUD/GUI object stays on screen. it all tracks the objects just fine just like you’d expect if it were tracking any other type of marker. IMO, object detection is marker-based detection - most people call marker-based detection what happens when you use a hiro, aruco, qr code, bar code or any kind of marker. In this case, marker = any object detected by OpenCV.

The issue I’ve run into is my math, I guess. Taking the OpenCV coordinate and turning that into something BabylonJS can use more readily. Is there a util library out there? Or do I need to take the examples for BabylonAR and pull out the OpenCV x and y and convert that over?

Also, I noticed that somewhere in my thousands of tabs open is an example of BabylonAR detecting markers with the webcam (which I also have) - so if possible I’d like to remove my custom code and use BabylonAR for that.

Like I said, right now, it does most of what I’m looking to do before I start adding more of my OpenCV code to the mix.

I can expand more about my project if that helps. After I get the OpenCV portion working the way I want then I’ll be working on fixing the SLAM work I’ve done.

I hope this makes sense.

Thanks

2

Well, I guess I just needed a rubber duck. The WebRTC code I am using is already in VideoTexture.ts with CreateFromWebCamAsync - I added my deviceId implicitly to the constraints and it works perfectly. I’ll use the example I found in BabylonAR to link up with the code I’ve already written and see how that works.

I’ll use this thread to update my progress through this thread.

1 Like
3

hey @winkdoubleguns it sounds great, would love to see your progress and demo when its available. Good luck

1 Like
4

Absolutely. I’ll put it up in GitHub.

1 Like