Working port to Rollup

[Symbitic]

I’ve completed a basic working port of Babylon.js ported to use Rollup instead of Gulp + Webpack. You can see my ongoing work at https://github.com/Symbitic/Babylon.js. When it’s complete, Gulp can be removed entirely, and Webpack can be used for just the web applications.

It’s not feature complete yet. It doesn’t build the web apps. It doesn’t build minified yet. It doesn’t handle linting or emitting typescript definitions. And it doesn’t bundle all the separate modules (Core, Gui, Loader, etc) into a single file for single distribution. I’m working on it.

But it does build both ES6 modules and UMD JavaScript for the Core, GUI, Loaders, Materials, PostProcess, ProceduralTextures, and Serializers. All in about 30 seconds.

Why use Rollup

Right now, Babylon is tightly integrated with its build system. That hampers maintainability.

Rollup generates much smaller bundles than Webpack. It can also handle building multiple formats at once.

It also means we can transition away from that Gulp + Webpack mess Babylon currently uses. Rollup builds with just one command: $ npm run build. And since Rollup is installed automatically, that means no more globally-installed packages.

The only two files used for the entire build process are rollup.config.js and rollup-plugin-babylon.js in the root directory. Try checking them out - I don’t even need to use the Tools/Config/ files.

Using

Just run npm run build. That’s it.

Files are placed in ./dist/rollup/. I added this directory to .gitignore, since I really don’t think repos should be committing built files - they should be using NPM and a CDN like UNPKG for that.

I have tested (though not extensively), and both the ES6 and UMD modules for Core can be loaded in Node.js, though the others (GUI, Loader) don’t work only because they require the “babylonjs” module installed (which it isn’t during development). The UMD modules for Core, GUI, Loaders, and all others load successfully in the browser and are available under the usual BABYLON.

Future

Obviously, I plan to add things like minified builds soon.

I hope to start porting things like linting, testing, type emitting and typedoc away from Gulp tasks and to NPM scripts, IF the developers are okay with that.

One thing I would like to do in the future is port to using the new Babel TypeScript preset. It offers faster compile times, it’s MUCH faster when running in watch mode, it doesn’t give you annoying error messages when you don’t want them. It has a preset for automatically transpiling to support even the oldest of web browsers. It allows us to use the very latest JavaScript syntax and plugins. I would post the links but I’m only allowed two links per post.

Help

I am hoping to submit a pull request soon, if the developers are okay with it. But first, I would like to get feedback.

Do people think this is a good idea to begin with? Is there something I should change?

I welcome any and all feedback.

[Deltakosh]

Pinging @sebavan who is the overseer of the build system

[sebavan]

As long as the emitted code/typings/maps are fully back compatible, if it is faster more maintanable and smaller it would definitely be a go for me.

That said, some of our code is not directly under BABYLON like the BABYLON.Debug namespace which was in before the migration. It is the same in the Loader for instance with nested namespaces for Gltf and so on…

Our es6 builds are currently only using tsc without any extra tooling needed and it would be cool if we could keep it as is as it means to build the es6 tools, you only need typescript without anything else.

We will only be able to merge when we reach feature parity between builds.

[Symbitic]

Fair enough.

What do you mean by the typings and maps being fully backward compatible? If you mean source code maps, Rollup generates those the same way Webpack does, so although they wouldn’t be the same byte-for-byte, they would still be used in the same way. Typings are always generated using TypeScript, whether you are running Rollup or Webpack. Right now, Rollup does NOT generate typings every time it runs (I just need to change declaration: false in rollup.config.js). So there shouldn’t be any difference between typing for Rollup or Webpack.

Just to make sure I understand what you are saying about the BABYLON namespace correctly, I checked and all of these are defined and usable: BABYLON.Engine, BABYLON.GUI.AdvancedDynamicTexture.CreateFullscreenUI, BABYLON.GLTFLoaderCoordinateSystemMode.FORCE_RIGHT_HANDED, BABYLON.DebugLayer, BABYLON.AxesViewer.

When I was making this I tried to port directly from Webpack to Rollup as much as possible. I even ported some parts of Gulp directly, like the Shader builder. Rollup now builds the TS files for all vertex and fragment shaders without ever writing a single file to the filesystem.

My goal is to eventually build everything (every single file) by just running npm run build in the root directory, and Rollup will generate all of those files directly. I may be wrong, but I thought that the ES6 modules were built by running Gulp (i.e. not by just running tsc directly). So technically that is an extra tool.

To help me, can you please give me a complete list of EVERY SINGLE FILE you expect to be generated by gulp typescript-all?

I realize it will need to acheive complete parity before it can become the default, but would it be possible to at least start merging before then? Right now, all generated files are placed in dist/rollup/, similar to what is done now with dist/preview release/.

When everyone is confident enough, I could change to only put certain files in dist/ in a staged transition (e.g. dist/babylonjs.serializers.* but still placing all other files in dist/rollup/). This way, could verify bit by bit that everything still works exactly the same, instead of one big transition at once.

[sebavan]

All the files in
dist/preview release/babylon.**
dist/preview release/gui
dist/preview release/inspector
dist/preview release/loaders
dist/preview release/materialsLibrary
dist/preview release/nodeEditor
dist/preview release/postProcessesLibrary
dist/preview release/proceduralTexturesLibrary
dist/preview release/serializers
dist/preview release/viewer

.temp/** containing our preview for npm installed with npm link to automate our delivery testing

Also all the tests ran during the build as well as our typedoc generation and playground typings generation

Gulp is also responsible to drive our local dev by generating and watching in es6 mode for core and the rest as bundles.

I guess a first step before starting to merge would be:

• babylon compiled to both es6 and umd
• BABYLON.Debug namespace respected (look for declare module BABYLON.Debug in https://raw.githubusercontent.com/BabylonJS/Babylon.js/master/dist/preview%20release/babylon.d.ts)
• .d.ts generation for both the umd version and the es6 version
• .map files generated
• local dev support from vscode as running today or better
• loaders library migrated the same way

What is the size comparison for the minified bundle at the moment ?

[Symbitic]

Status:

• Babylon and all other modules compiled to both UMD and ES6.
• BABYLON.Debug namespace added (in max, still working on regular).
• .map files generated.
• .d.ts generated (not entirely sure what the difference is between the UMD and the es6 version is supposed to be)

I don’t have minified working yet, but for comparison, here’s babylon.gui.js

• Webpack: 678kb
• Rollup: 578kb

Rollup completes babylon.gui.js (both the ES6 and UMD version) in about 13 seconds total. How does that compare with the current build time?

My plan is to integrate Rollup into the Gulp process. Basically, replace all Gulp calls to Webpack with Rollup.

But before I proceed any further: how many of the Babylon.js developers actually want this?

I started this project because I noticed that the build system used by Babylon was big and showing signs of age. I wanted to make the process of contributing to Babylon easier, and I thought that transitioning to Rollup would help. Rollup would complete faster (versus Webpack), and it would generate smaller bundles. And eventually, it might even be possible to transition away from using gulp at all (e.g. using npm run build to build using rollup, npm run tslint to lint, npm run distribute to upload to netlify, etc). It would go a long way towards mitigating some of the technical debit from such a large (and in certain areas legacy) codebase.

But because I have to be feature complete with EVERYTHING before I can merge, this is requiring an awful lot of effort. So I would like to know if there is anyone besides me that actually wants this completed (and preferably is willing to help me). Because if not, I will leave the process of maintaining and distributing Babylon to the existing developers.

[aWeirdo]

Does Rollup generate anything like Webpack’s

" XX__WEBPACK_IMPORTED_MODULE_0__[“someClass”] "

in the js files`? ( used to just be BABYLON.someClass pre-4.0)
or any other simular changes?

That is something which has bugged me, makes it tedious to copy/paste source code to quickly test some changes in the PG, etc.

[Symbitic]

No. Rollup was designed to bundle cleanly from the ground up.

Thats why it generates smaller bundles than webpack. The word “ROLLUP” doesn’t appear inside the bundled source code anywhere.

[aWeirdo]

then i like it already.

[Symbitic]

Current list of generated files:

dist/rollup/babylonjs.gui.js
dist/rollup/babylonjs.gui.js.map
dist/rollup/babylonjs.gui.min.js
dist/rollup/babylonjs.gui.min.js.map
dist/rollup/babylonjs.gui.mjs
dist/rollup/babylonjs.gui.mjs.map
dist/rollup/babylonjs.gui.module.d.ts
dist/rollup/babylonjs.js
dist/rollup/babylonjs.js.map
dist/rollup/babylonjs.loaders.js
dist/rollup/babylonjs.loaders.js.map
dist/rollup/babylonjs.loaders.min.js
dist/rollup/babylonjs.loaders.min.js.map
dist/rollup/babylonjs.loaders.mjs
dist/rollup/babylonjs.loaders.mjs.map
dist/rollup/babylonjs.loaders.module.d.ts
dist/rollup/babylonjs.materials.js
dist/rollup/babylonjs.materials.js.map
dist/rollup/babylonjs.materials.min.js
dist/rollup/babylonjs.materials.min.js.map
dist/rollup/babylonjs.materials.mjs
dist/rollup/babylonjs.materials.mjs.map
dist/rollup/babylonjs.materials.module.d.ts
dist/rollup/babylonjs.max.js
dist/rollup/babylonjs.max.js.map
dist/rollup/babylonjs.max.mjs
dist/rollup/babylonjs.max.mjs.map
dist/rollup/babylonjs.min.js
dist/rollup/babylonjs.min.js.map
dist/rollup/babylonjs.mjs
dist/rollup/babylonjs.mjs.map
dist/rollup/babylonjs.module.d.ts
dist/rollup/babylonjs.postProcess.js
dist/rollup/babylonjs.postProcess.js.map
dist/rollup/babylonjs.postProcess.min.js
dist/rollup/babylonjs.postProcess.min.js.map
dist/rollup/babylonjs.postProcess.mjs
dist/rollup/babylonjs.postProcess.mjs.map
dist/rollup/babylonjs.postProcess.module.d.ts
dist/rollup/babylonjs.proceduralTextures.js
dist/rollup/babylonjs.proceduralTextures.js.map
dist/rollup/babylonjs.proceduralTextures.min.js
dist/rollup/babylonjs.proceduralTextures.min.js.map
dist/rollup/babylonjs.proceduralTextures.mjs
dist/rollup/babylonjs.proceduralTextures.mjs.map
dist/rollup/babylonjs.proceduralTextures.module.d.ts
dist/rollup/babylonjs.serializers.js
dist/rollup/babylonjs.serializers.js.map
dist/rollup/babylonjs.serializers.min.js
dist/rollup/babylonjs.serializers.min.js.map
dist/rollup/babylonjs.serializers.mjs
dist/rollup/babylonjs.serializers.mjs.map
dist/rollup/babylonjs.serializers.module.d.ts

Current build time: 3 minutes

For babylon.max.js:
Webpack = 7.6 MB
Rollup = 2.9 MB

Biggest thing left is watch mode and the other .d.ts declarations besides the modules.

mjs files are ES6 modules (that’s technically the correct extension for ES6 modules).

I could be ready in just a few days.

[sebavan]

I wonder what is impacting 5 Mb in the code, it is probably something we would need to double check as it sounds like a lot

About the .d.ts, the files are totally different between modules/non modules and es6. For this part we could probably keep the current process.

I ll try to see the biggest diff on your branch.

I would be all in if that brings faster and smaller builds

[Deltakosh]

I’m all in if we save file size!!!

[RaananW]

I remember trying rollup a year ago (trying to create a better es6 version), but back then it was impossible because babylon was not module-based. I think the idea is wonderful, but as @sebavan said, the 5MB difference is certainly not webpack overhead (which, i assume, is ca. 300kb al in all?)

[sebavan]

yup my rollup tests last year were about 200kb but had issues with some of the typescript dependencies to bundle correctly. The speed difference was about 5s per lib but the memory was also higher during the build.

I will test the branch locally to spot the biggest diff today.

[sebavan]

@Symbitic, I still wonder why not using purely tsc for es6 ? What does rollup bring here ?

[chase-moskal]

@sebavan

@Symbitic, I still wonder why not using purely tsc for es6 ? What does rollup bring here ?

i’m also wondering the same thing – why does babylon have a complicated build setup at all? and why does babylon feel responsible to provide a bundle?

as an observer in the past, i was completely overwhelmed by the complexity of the babylonjs build, and it caused me to go work on other things instead of contributing

as tech and workflows in this industry advance, modern packages are becoming simpler and simpler in terms of build complexity, which is a really good trend

it would be most ideal if babylon’s core build could be reduced to not much more than a simple tsc call, and emit es modules only

• consider how modern packages like lit-element only provide simple es modules (no bundles, no minification)
• the modern best practice is to leave the bundling/minification/optimization concerns to the users or to recommend services like the fantastic new pika cdn which creates bundles on-demand, look at this instant-bundle for lit-element: https://cdn.pika.dev/lit-element
• if you’re not peeing your pants in excitement over that last point, please reread it
• this allows packages to be lean and only contain simple es code, and still allows everyone to have the bundles and performance that they want

let’s leave the minification/bundling/rollups/wepacks in the hands of your users, otherwise they can (and should!) easily use a convenience service like that amazing pika bundler – this is how cutting-edge packages are doing it, and it’s working fantastically today – and these modern projects enjoy very very little build complexity, it’s generally just a simple tsc call and that’s the end of the story

• babylon’s main module entrypoint (in the package.json) should be changed to simple es module – not a bundle as is currently the case – i see the current bundle-first as a bad practice and is not in-line with the future ecosystem that pika and other tools are pioneering
• it’s still a good idea to provide UMD modules and also some bundles, but deprecated and for legacy workflows

please let me know what you folks think of these emerging practices, cheers!

Chase

[sebavan]

This is exactly what we are doing in our scoped package: “@babylonjs/…” like @babylonjs/core - npm

When developping locally, core is actually only loaded directly from es6 in the browser.

But we do not want to let down the community and their habits of using our bundled version.

This is the only and main reason we keep maintaining our bundle as backward compatibility is key for us. The new es6 relies exclusively on tsc for map d.ts and transpiling and the user can then consume it as they want.

Our bundle entry points are actually called legacy : Babylon.js/src/Legacy at master · BabylonJS/Babylon.js · GitHub for this reason.

But it does not prevent to improve the bundle in parallel if there is a huge added value.

Here the only one I see is actually rollup scope hoisting saving a bit of bytes in the lib and helping the first load time.

[sebavan]

@Symbitic here are the first results I collected:

• The file size is different by 100k due to scope hoisting in rollup prevented by global use in webpack but I might be able to trick this one. (your 5 Mb comparison is based on max vs min (coming from your extension detection in your rollup config))
• The build time differs by only 1 second
• The code does not work anymore so far when running it in the browser
• The esm version should not be bundled to let the client code works normally
• The .d.ts seem broken at the moment, looks like they emit in the same file

Let me know if you need anything ?

Maybe we could only use rollup for the main bundle but it would still be quite an investment to reduce 20k gzipped knowing you should rely on our es6 package if you are into a smaller footprint.

Any thoughts ?

[chase-moskal]

@sebavan

This is exactly what we are doing in our scoped package: “@babylonjs/…” like @babylonjs/core - npm

oh hey that’s great! i dig it! i keep finding my way to the old babylon package/repo and getting confused… can that legacy stuff just be deleted already? who knows how much confusion and havoc it continues to wreak on a daily basis

edit: where even is the sourcecode for @babylonjs/core? i can’t seem to find it… it doesn’t appear to be a branch of this repo

given this discussion, i would not be in favor of the addition of a rollup bundle, or any bundle for that matter – all bundles should be deprecated and no new ones should be created

thanks @sebavan bringing me up to speed on the current state, clearly some really good work has been done to simplify babylon!

cheers

[sebavan]

the sources are the same we just have this step of our build:

BabylonJS/Babylon.js/blob/master/Tools/Gulp/tasks/gulpTasks-librariesES6.js#L191

creating the packages before we push them to npm.

And basically the result packages look like :

allowing any bundler to consume it.