Points Cloud System and KITTI vision (Velodyne laser scanner autonomous driving dataset)

KITTI dataset vision benchmark suite and more information about that project --> http://www.cvlibs.net/datasets/kitti/index.php.

I wanted to integrate AssetManager into react-babylonjs and support React Suspense. Suspense is the newer and still “experimental” way to load data, which has a fallback component render while data is loading before rendering the component that displays data/model/texture/etc. - Suspense provides a better user experience and easier to follow code. babylonjs handles that generally by having a loading UI that can be customized. I have model demos that show a 3D progress bar as fallback, but unfortunately reporting progress doesn’t look possible with Suspense (without context). Anyway, I first created a useAssetManager hook to load assets via AssetManager:

const pointCloudAssets = [{ taskType: TaskType.Binary, url: 'assets/kitti/000000.bin', name: 'KITTI' }];

const MyComponent = () => {
  const [result] = useAssetManager(pointCloudAssets);
  ...
  return <pointsCloudSystem name='kitti-points-cloud' pointSize={2} />
}

Above code loads the binary file with cloud points data, but it needs to be wrapped in a Suspense component (or will trigger an ErrorBoundary) like this:

<Suspense fallback={<MyFallback />}>
  <MyComponent />
</Suspense>

The first part to build this was easy - just wrap AssetManager into a promise using callbacks:

const assetManager: AssetsManager = new AssetsManager(scene);
// add tasks here

const taskPromise = new Promise<AbstractAssetTask[]>((resolve, reject) => {
    let failed = false
    assetManager.useDefaultLoadingScreen = useDefaultLoadingScreen;
    assetManager.onFinish = (tasks: AbstractAssetTask[]) => {
        if (failed === false) {
            resolve(tasks);
        }
    };
    assetManager.onTaskError = (task: AbstractAssetTask) => {
        failed = true;
        reject(task.errorObject?.message)
    };

    assetManager.load();
});

The weirdest thing about Suspense is that you need to throw that promise (yes, throw like you throw an error). This in turn causes your actual component to not render, so you cannot use hooks like useMemo or useState as they do not persist and you will end up in an endless loop - I memoized the exported hook. So, that was probably the main confusing part. You can do so like this:

const useAssetManagerWithCache = () => {
    // our own memoized cache
    const suspenseCache: Record<string, () => Nullable<AbstractAssetTask[]>> = {};
    return (tasks: Task[], useDefaultLoadingScreen: boolean = false, scene?: Scene) => {
        if (suspenseCache[key] === undefined) {
            // code in here to promisify the AssetManager (as above) as a function that will
            // throw the promise before completion or return data/error
            suspenseCache[key] = createGetAssets(tasks);
        }
    }

    const fn: () => Nullable<AbstractAssetTask[]> = suspenseCache[key];
    return [fn()];
}

export const useAssetManager = useAssetManagerWithCache();

So, the fallback looks like this with a spinning cube - or whatever to indicate loading:

Then when the binary cloud point data loads, the regular component loads:

I didn’t put these changes out on NPM yet, but they are on master :slight_smile: Still deciding if this is how I want to roll it out. Maybe it makes sense to put this in the lightweight babylonjs-hook NPM instead (and also move the context stuff out as well). Just experimenting. Here is a working demo:

3 Likes

Excellent!!

@Necips I got a message that you had asked a question here (that was removed?) for the point cloud data for a ray marching algo The raw file is here:
react-babylonjs/000000.bin at master · brianzinn/react-babylonjs (github.com)

I used the asset manager to load binary that file and then this was how the data is loaded into position Vector3’s:

const floats = new Float32Array(assetManagerBinaryTask.data);
const POINTS_PER_FLOAT = 4;
const numPoints = floats.length / POINTS_PER_FLOAT;

// particle is the current particle, the i-th one in the PCS and the s-th one in its group
const particleFunc = (particle, i, s) => {
  // KITTI-formatted PCD
  const x = floats[POINTS_PER_FLOAT * i]
  const y = floats[POINTS_PER_FLOAT * i + 1]
  const z = floats[POINTS_PER_FLOAT * i + 2]
  // ignore the reflectance value
  // const r = floats[POINTS_PER_FLOAT * i + 3]
  particle.position = new Vector3(x, y, z)
  particle.color = Color4.FromColor3(Color3.White())
}

// pcs is the PointCloudSystem...
pcs.addPoints(numPoints, particleFunc);
pcs.buildMeshAsync(() => {
  setPcsMesh(pcs.mesh);
})

You shouldn’t get a JSON parse error - make sure you are using the binary task that returns an ArrayBuffer:
Asset Manager | Babylon.js Documentation

If you can’t figure it out from there I can make a playground. The other thing I wanted to mention is that you may want to use the reflectivity of the points, which I am discarding. I suspect it may be useful to provide a normal (when the reflectivity is very high),…

@Necips 've seen some good Random sample consensus (RANSAC) make good meshes of point clouds. Can create some nice reconstructions. Would be really cool to use on these new phones with lidar :slight_smile: