Investigate Scala 2 community build

[romanowski]

How scala 2 handle:

• a project with 2 different versions
• native vs js
• running all tests projects?
• is there any mechanism to make sure that all deps are translated
• what found most problems?

[SethTisue]

a project with 2 different versions

Well first of all, often it doesn't matter as long as the changes made were source compatible, and a surprisingly high percentage of the time, that's actually the case.

But of course sometimes two versions aren't source compatible, and there's a problem.

I'll answer for how things were before https://contributors.scala-lang.org/t/scala-2-13-community-build-radically-simplified/5244 , since the simplification we made there was only possible because the Scala 2 ecosystem reboot for 2.13 was the last Scala 2 reboot ever.

Before the simplification, we almost always forbid having multiple versions of things. This often created difficulties, and that's where a lot of the maintenance effort went. For example, Cats Effect 2 versus Cats Effect 3. We didn't want to support both, so we stayed on 2 for a long time, freezing downstream projects whenever they upgraded to 3. Then eventually almost everyone had upgraded, so we moved to 3, unfreezing all that stuff we froze, and if any stragglers were left behind on 2, we leaned on the maintainers to upgrade, and if they didn't or wouldn't, we considered the project abandoned and we removed it.

That said, there were times where it was absolutely necessary to support more than one version of something. For example, for a long time we supported both ScalaTest 3.0 and 3.2. dbuild doesn't allow duplicate artifacts (same org + name, regardless of version number) in the same "space", so we forked 3.0, changed the org name in the fork, and then in any project that used 3.0, we overrode the dependency to use our org name instead.

dbuild also offers a "spaces" feature where you can have multiple versions of things as long as each one is in a different space. But that meant partitioning the build into separate subtrees. This was sometimes possible and useful in specific circumstances. For example, for a long time we had a jawn 0.10 and a jawn 0.11 space (both of which depended on a shared space). The jawn 0.10 space just had sbt and a few other things in it, so it was manageable to keep that separate. This kind of splitting was possible, but we always tried very hard to avoid it because it complicated maintenance and because it didn't scale to having more than a very small number of version splits.

(And what about after the radical simplification? We still require a single version only of a handful of core dependencies such as kind-projector and scalameta, but everything else is a free-for-all and it's fine, since everything else uses binary dependencies from Maven Central.)

native vs js

This was out of scope for us, partly because changes to dbuild would have been needed (and only Toni knew the codebase), and partly because this was a Lightbend thing and Lightbend is almost exclusively JVM focused (as a company generally, and within our team specifically, too).

running all tests projects?

This happens by default with dbuild. If you don't want a subproject included, or don't want to run some tests, you have to explicitly exclude them.

is there any mechanism to make sure that all deps are translated

You mean to ensure that everything is actually built from source? Yes, dbuild enforces that with check-missing: true which was the default before the radical simplification.

what found most problems?

So we have 200+ projects and I half wish I could tell you that it was wasted effort and it was only a certain subset that found problems.

But my actual experience is that having so many projects was actually worthwhile. I found it entirely unpredictable what kind of project a Scala regression might pop up in, and it astonished me how often only 1 or 2 projects in the entire build would notice a regression, and how often those 1 or 2 projects would be something random rather than a "core" project.

In short, what found most problems was to have a HONKING BIG BUILD.