On the road to 1.0

[chrispcampbell]

I'm happy to report that we at Climate Interactive will be contributing (as open source) more of the software libraries that we use to produce our popular En-ROADS and C-ROADS simulators. We are planning to migrate the source code for these libraries from our internal repos to the public SDEverywhere repo on GitHub over the coming weeks.

First, a quick backstory. For the past few years, Climate Interactive has been using the SDEverywhere transpiler, developed by Todd Fincannon, to translate the En-ROADS and C-ROADS system dynamics models into a form (WebAssembly) that can run extremely fast in any web browser. The SDEverywhere tools are key in making the En-ROADS and C-ROADS web simulators what they are today.

After I joined Climate Interactive in June 2020, I started contributing fixes and improvements to the core SDEverywhere transpiler, but I also began building (in collaboration with Travis Franck) more software packages around the core of SDEverywhere. These packages aimed to make it easier to transform any system dynamics model (working in Vensim) into modular, web-friendly components. Among other things, we've developed:

• a build system on top of the core SDEverywhere that simplifies the process so that anyone can type one command and have a working model running in the browser in a matter of seconds
• an extensive "model-check" tool that allows you to run automated checks that can verify the behavior of a model (similar to unit/integration tests in the software development world) and compare to earlier versions of the model

We feel that these tools fit comfortably within the scope of the SDEverywhere project. With that in mind, my plan is to change the SDEverywhere repo from holding a single monolithic sdeverywhere package into a monorepo that holds many packages under the SDEverywhere umbrella. Initially, like the current sdeverywhere package, these will be given 0.x version numbers, but once the packages settle, we plan to declare 1.0 versions of all of them and hope to publish releases more regularly than we have in the past.

Here's a summary of what I've done so far, and what I'm planning for the coming weeks:

1. I took what we had in the repo and published it as [email protected] (finally). That way we were able to close the long-under-development 0.6 milestone and move on. That should be our last release of the "classic" sdeverywhere package. (I also published the [email protected].)

2. I converted the SDEverywhere repo to have a monorepo structure, and moved the existing sdeverywhere package files. I had been using yarn for our internal monorepos, but switched to pnpm a while back and have found it to be a better (and faster) replacement, so that's what we'll be using for the SDEverywhere repo.

3. I set up release-please to help manage publishing the new packages. I've been very happy with this; it automates and simplifies the release process, which is especially important when there are multiple packages involved.

4. Now that those prep steps are done, I will be migrating packages from our internal repos over to the SDEverywhere repo. We recently secured the @sdeverywhere npm org name, so I'm planning to publish new packages under that name to keep everything under the same logical umbrella. I'm going to publish these with versions starting at 0.1.0, and once the dust settles, we can publish 1.0 versions and take things from there.

Here's the package layout I'm planning (all of these names will be prefixed with the @sdeverywhere org):

• compile
• build
• cli
• runtime
• runtime-async
• check-core
• check-ui-shell
• plugin-wasm
• plugin-worker
• plugin-vite
• plugin-check
• kit

A brief overview of the packages:

• The existing sdeverywhere package will basically be migrated into the compile and cli packages. I'm not planning to do any major refactoring of those existing sources right away; they'll remain JavaScript and have the same internal structure they do now for the most part. (All other packages noted below are developed in TypeScript.)

• The runtime and runtime-async packages help make the WASM model (produced by SDEverywhere transpiler and Emscripten) easy to work with in a JavaScript-based library or application.

• The check-core and check-ui-shell packages are basically the thing we call "model-check" (the model check and comparison suite).

• The build package is what makes for both a pleasant local development experience and for continuous integration builds (in GitHub Actions, for example). The plugin packages allow for customizing and configuring the build process.

• And finally, the kit package is meant to be a single convenient entry point to SDEverywhere for most users; it includes all the other packages as dependencies.

The end result for all of this is that users should have a simple (but powerful) "out of the box" experience. Given a Vensim mdl file, the user should just be able to do the following (note, no config files needed initially):

# Install one package
npm install @sdeverywhere/kit

# Build a js/wasm model
sde build sample.mdl

# Build a js/wasm model and run model-check
sde build --check sample.mdl

# Run sde in "dev" mode (with "live reload" enabled), which opens a live
# "explorer" app plus a live model-check report
sde dev --check sample.mdl

# It should still be possible to use sde in its more minimal form, similar
# to how it can be used today to generate C code only, for example
sde generate --c sample.mdl

For more fine-tuning, the build process (and the individual plugins) can be configured using an sde-config.js file. Our whole En-ROADS and C-ROADS build processes have been rewritten to use the new build/plugin packages, so all the things that have worked well for us internally (and all the complexity/drudgery I've worked through and papered over) should now work equally well for anyone else's projects.

Stay tuned as the pieces fall into place. We are hoping to make more use of this discussion board, so please feel free to post any questions or suggestions.