Points Cloud System and KITTI vision (Velodyne laser scanner autonomous driving dataset)

KITTI dataset vision benchmark suite and more information about that project --> http://www.cvlibs.net/datasets/kitti/index.php.

I wanted to integrate AssetManager into react-babylonjs and support React Suspense. Suspense is the newer and still “experimental” way to load data, which has a fallback component render while data is loading before rendering the component that displays data/model/texture/etc. - Suspense provides a better user experience and easier to follow code. babylonjs handles that generally by having a loading UI that can be customized. I have model demos that show a 3D progress bar as fallback, but unfortunately reporting progress doesn’t look possible with Suspense (without context). Anyway, I first created a useAssetManager hook to load assets via AssetManager:

const pointCloudAssets = [{ taskType: TaskType.Binary, url: 'assets/kitti/000000.bin', name: 'KITTI' }];

const MyComponent = () => {
  const [result] = useAssetManager(pointCloudAssets);
  ...
  return <pointsCloudSystem name='kitti-points-cloud' pointSize={2} />
}

Above code loads the binary file with cloud points data, but it needs to be wrapped in a Suspense component (or will trigger an ErrorBoundary) like this:

<Suspense fallback={<MyFallback />}>
  <MyComponent />
</Suspense>

The first part to build this was easy - just wrap AssetManager into a promise using callbacks:

const assetManager: AssetsManager = new AssetsManager(scene);
// add tasks here

const taskPromise = new Promise<AbstractAssetTask[]>((resolve, reject) => {
    let failed = false
    assetManager.useDefaultLoadingScreen = useDefaultLoadingScreen;
    assetManager.onFinish = (tasks: AbstractAssetTask[]) => {
        if (failed === false) {
            resolve(tasks);
        }
    };
    assetManager.onTaskError = (task: AbstractAssetTask) => {
        failed = true;
        reject(task.errorObject?.message)
    };

    assetManager.load();
});

The weirdest thing about Suspense is that you need to throw that promise (yes, throw like you throw an error). This in turn causes your actual component to not render, so you cannot use hooks like useMemo or useState as they do not persist and you will end up in an endless loop - I memoized the exported hook. So, that was probably the main confusing part. You can do so like this:

const useAssetManagerWithCache = () => {
    // our own memoized cache
    const suspenseCache: Record<string, () => Nullable<AbstractAssetTask[]>> = {};
    return (tasks: Task[], useDefaultLoadingScreen: boolean = false, scene?: Scene) => {
        if (suspenseCache[key] === undefined) {
            // code in here to promisify the AssetManager (as above) as a function that will
            // throw the promise before completion or return data/error
            suspenseCache[key] = createGetAssets(tasks);
        }
    }

    const fn: () => Nullable<AbstractAssetTask[]> = suspenseCache[key];
    return [fn()];
}

export const useAssetManager = useAssetManagerWithCache();

So, the fallback looks like this with a spinning cube - or whatever to indicate loading:

Then when the binary cloud point data loads, the regular component loads:

I didn’t put these changes out on NPM yet, but they are on master :slight_smile: Still deciding if this is how I want to roll it out. Maybe it makes sense to put this in the lightweight babylonjs-hook NPM instead (and also move the context stuff out as well). Just experimenting. Here is a working demo:

2 Likes

Excellent!!