Scala Center Proposal: Community Build

[ghost]

I was chatting on the the scala/community-builds gitter about community builds and the scala center(SC). This particular link was regarding a scala.js community build.

So this issue it about getting more SC participation in this area. Right now, I'm not totally sure what that should be specifically - but I think there is a natural fit here.

But I'll make a start by saying that the scala community build is really for the benefit of scalac - fair enough, it is theirs. I can see why scala.js would want one too.

But many others projects could benefit from their own, too (eg typelevel) . So perhaps a goal of the SC could be to promote the development and use of community builds in general - and this would also nicely tie in with their scaladex.

cc/ @sjrd and @SethTisue as they were on the chat

[tpolecat]

Can someone with sufficient sparkles create a Scala Center label?

[milessabin]

✨

[larsrh]

Agreed in principle. But in order to turn this into a Scala Center proposal we would need to specify some actionable items. Does anyone have something in mind? Off the top of my head: spend some work to make it easy for third parties to curate their own community build. Can we go from there?

[ghost]

A few tweaks to catalysts-docker might well be enough to get the practical side going. The documentation is clearly missing there, but catalysts-docker + dbuild should be enough to get local community builds(CB) going.

The bigger problems stem from when one actually starts to run the CB - esp cyclic/missing dependencies. I'll add a post here later on that....

[ghost]

@SethTisue Don't suppose you know anyone that has any thoughts on the practical aspects of running a community build?

[ghost]

Continuing from earlier:

A few tweaks to catalysts-docker might well be enough to get the practical side going.

I think that once the concept is promoted a bit and a few more people actually start doing such builds, a raft of other issues will become apparent, in particular at the "low-end" of the stack where any problems also have the most impact.

By example (almost real..):

Cats is a library that many libs depend on, and it depends on scalatest (and others). scalatest would like to use cats, but can't, because of the cyclic dependency .

But hey... the cats dependency is only a test dependency on scalatest so it's OK. Ahh, but how do we release cats without testing it? And if we hack it, how do we tell dbuild to hack it?

So one day, someone splits cats into like 12 repositories and re-configures dbuild. Sametime, half the cats maintainers leave as they can no longer work in an environment that is geared towards distributive builds rather than abstract coding.

So the twelve repositories are merged back, into two repos.

And scalatest copies and pastes the bits of cats it needs into scalatest to avoid it's dependency issue, and then....
scala 2.12.0-RC2 is released and then....

I think the real value that the SC can add (in the short term) is the promotion of community builds and collecting the common issues, that will arise from many projects

Given that it looks like the SC are currently "stretched", having a proposal that is more community centric rather than, say, coding centric is useful as :

• the SC are less unlikely to park it.
• the board are less unlikely to reject/park it
• the community are more likely to accept that the SC is moving to encompass them

Also, most initial work would likely come from scalac, scala.js, typelevel - hence allowing the SC to really focus on the soft part.

[sjrd]

scalatest would like to use cats, but can't, because of the cyclic dependency .

IMO the story simply stops there. It can't, period. Do not introduce cyclic dependencies, even if it's "just" in your Test config.

That's why Scala.js uses JUnit and not ScalaTest or any other Scala dependency. Doing anything else is madness.

[SethTisue]

Don't suppose you know anyone that has any thoughts on the practical aspects of running a community build

what would you like to know? I can certainly confirm there are "practical aspects". it's the single thing I've spent the most time on.

I recently did a pass through the documentation at https://github.com/scala/community-builds/wiki and updated everything, so that's a good source of information.

Jenkins requires care and feeding (https://github.com/scala/scala-jenkins-infra). somebody has to pay the AWS bills, too (thx, corporate overlords ✨). the interaction with PR validation on GitHub (https://github.scom/scala/scabot) takes care and feeding. all of this stuff is on GitHub and available for raiding, but adapting or repurposing it for other orgs needs besides just the Scala compiler team's would be work, of course.

dbuild was a great deal of work to create (thx again 💵✨). but you should know that there is zero or nearly-zero budget for ongoing development, at least at the moment. the Scala community build uses dbuild 0.9.5, it works fine, if we found a bad bug probably @cunei would find some spare moments to fix it for us, but in your planning, you might want to take into account that you can't assume that dbuild is actively being developed and improved. it isn't. Lightbend used to use it to build the Lightbend Reactive Platform, but we don't anymore as of RP 2, so that's no longer a driver.

as far as whether this ought to be a Scala Center priority: I like the Scala.js-community-build idea. I'm not sure there are other problems this solves?

[SethTisue]

here is a concrete suggestion for Typelevel: we would have love to have more Typelevel projects in the Scala community build. I've made some progress on it in the last year-plus. but adding them is work. quite apart from whether y'all decide to do your own community-build-like-thing, It would be fantastic if the Scala community build itself had higher visibility within Typelevel and if it were encouraged and considered normal for library authors to add their libraries to the community build. I've done my best to document what's involved. it's usually necessary for me to do some handholding since library authors aren't familiar with dbuild, etc, but even with handholding, it's a lot less work for me than doing the whole thing myself.

we're no longer accepting additions to the 2.11 community build; here is the 2.12 projects list: https://github.com/scala/community-builds/blob/2.12.x/project-refs.conf

why is this beneficial for Typelevel, and not just beneficial for the Scala team? well, we'd like to think that anything that helps us helps y'all as well. but also, your projects push scalac pretty hard. we do our best not to break your stuff, but, also we can't always know in advance whether changes we're making might adversely impact your projects. if your code is in the community build, we know right away (sometimes even before a Scala PR is merged — we don't run the community build against every PR, but we run it against the tricky ones). so it's insurance for you that the things you're doing with Scala will continue working

[SethTisue]

Do not introduce cyclic dependencies

+1. dealing with the cyclic dependencies between scala and scala-xml, and scala and scala-parser-combinators (fixed in 2.12), and scala and scala-partest, and scala and scalacheck, consumes an inordinate amount of thought and effort. it's so much better not to get into this kind of trouble in the first place

[ghost]

IMO the story simply stops there. It can't, period. Do not introduce cyclic dependencies, even if it's "just" in your Test config.

Yep, that's the point!! I know that, you know that.. but the question is - where are the cyclic dependencies in the whole OSS community?. I think we know where they are at the "bottom" and I recently did a PR for new scoverage core that just uses junit.

But I did that because "I knew there was an issue" - but technically, someone else, or a process, could have done so. And the same process could work on many projects, for example from scaladex

[ghost]

Doing anything else is madness.

Another 👍 I'm sure there is plenty of madness in the scala OSS community as a whole, and that is where perhaps the SC can help: not to solve the issues, but highlight them somehow (for the more popular projects)

[larsrh]

What's the status of this? Next AB meeting has been scheduled; deadline for proposals is in about a week.

[ghost]

status:
typelevel/catalysts-docker#2
scala/community-build#474
scala/community-build#478

[ghost]

The changes to the setup ^^ allow more than one community build to exist in the subdir configs and enable different resolvers and project refs to be used. If accepted, the next step might be to split the project configuration up more.

So we can have, say, a typelevel CB, broken down such that the scala CB can import the parts it wants. This is more the just "DRY" - it means that as TL can just make sure it's own projects work, the scala CB can focus on their priorities.

And there is no need for all the CB's to live in the same repo either. So the goal here is to have more CB's, that can share configs as needed.

[milessabin]

This is great stuff ... thanks! I'll try and find time to give this a spin this week :-)

[larsrh]

I think we should continue discussion & work out how to incorporate it into a proposal at scalacenter/advisoryboard#20.

[SethTisue]

this came up again on #fs2-dev on the Typelevel Discord. I wrote the following, leaving it here for posterity:

How much maintenance time goes into community build weekly? Ballparks are cool.

I'm going to give a long, nuanced answer to this, since I fear a short one would be misleading.

The reputation the Scala community build has for being difficult to maintain is both deserved and undeserved. It is deserved during the run-up to a binary-incompatible Scala release, e.g. the run-up to Scala 2.13.0. All the maintainers are updating their code like crazy, and often doing it on experimental-ish branches rather than a stable-ish branch and thus, keeping the community build going while that's happening is pretty hard.

But during a binary-compatible phase like Scala 2 is now permanently in, maintaining the build isn't so hard. I bump the project SHAs every two weeks and sort out any incompatibilities that arise, make little adjustments due to upstream build changes, etc. And I bump the Scala SHA every two weeks or so also, and look into any breakages. It averages out to, idk, 2 or 3 hours a week.

Every so often there is a spike when some major upgrade to some important library needs to be pushed through. So for example the ScalaTest 3.0->3.1->3.2 upgrades have been pretty hairy to push through. Another example is that the Scala 2 community build is still on Cats Effect 2. Eventually I'll want to move it to CE3, and that might take a few person-days of effort (spread out over weeks, probably, since the build takes so long to run, so turnaround time for testing changes is slow).

Note that the community build has more than 200 projects in it. A Typelevel-only version would be considerably smaller and thus, easier to maintain, especially since Typelevel projects tend to be actively- and well-maintained and have clean builds.

Note that I would probably not suggest using dbuild itself for this, except perhaps as an experiment, because dbuild is no longer maintained. Soon we will stop using it for Scala 2.13.x, as per scala/community-build#1456, and the Scala 3 people are highly unlikely to adopt it, since it's unmaintained and the codebase is pretty hairy and Toni (the author) is largely unavailable.