We Don't Run Tests on a Continous Integration Server

[aaronjensen]

https://github.com/aaronjensen/software-development/blob/master/continuous-integration.md

[sbellware]

https://github.com/aaronjensen/software-development/blob/master/continuous-integration.md?plain=1#L6

Not CruiseControl in 2000 or CruiseControl .NET in 2003?

[aaronjensen]

Oh sheesh, you are right. We did use CruiseControl. We had a homegrown thing at Microsoft too, but I think the first one I set up was CruiseControl. It was still around 2006 though -- that's when I left MS.

[sbellware]

https://github.com/aaronjensen/software-development/blob/master/continuous-integration.md?plain=1#L45

Might be useful to point out that the team's norms reside within each and every member of the team, from the most junior to the most junior; that our culture and practices are tailored specifically to create and reinforce the ready knowledge of the norms; and our process compensates for the time required to teach and support team members because it's infinitely faster to not make mistakes than to be caused to become aware of them and then to correct them. Most teams will have never experienced this speed, and some won't even believe that it can be achieved. Often this is because their practices presume after-the-fact detection of mistakes and the ownership and operation of the tooling that does the detection and alerting. Rather than invest in mistake detection tooling and operations, we invest in the people, and count on them for mistake proofing. Or as Shigeo Shingo refered to it as part of the Toyota Production System: Poka-yoke. There's an exponential cost increase in managing a mistake the more time there is between making a mistake and correcting a mistake. The cumulative effects of mistakes eats more of a software team's productivity than they know - and most aren't even paying attention at this level. The ultimate in cost reduction and productivity enhancement is empowering the software developer to make fewer mistakes, or no mistakes at all. And while mistakes will always be a possibility, optimizing culture and process around eliminating mistakes pays far greater dividends than mere mistake-detection automated tooling.

[aaronjensen]

I'm adding more about this, but leaving this here as useful context for anyone that happens upon these discussions.

[sbellware]

https://github.com/aaronjensen/software-development/blob/master/continuous-integration.md?plain=1#L49

The point about attention can't be stressed enough. It's cultivation of attention that our process is principally concerned with.

[pengubco]

Thanks for sharing! I agree most of it because I've seen the cost of CI in real work.
How do you think about using CI to manage merging PRs to main? Different PRs each pass tests, but the merged commit may not even compile.

[aaronjensen]

Thank you.

Could you say more about what you mean with regard to using CI to manage merging PRs? Do you mean that they will automatically merge if the CI passes? Or do you mean that some sort of queue is used (like bors)?

If someone is running tests on a branch and then merging that branch into the primary branch without verifying that the work is still valid there, that's effectively negligence. It wouldn't be allowed on our team.

[pengubco]

In my case, the main branch is locked. Only few owners and Github can merge to main. PR is not automatically merged. PR author type bors merge to merge. Bors makes sure the merged commit passes tests.

I'd like to add another complaint of CI systems (Circle CI and Github). They do a bad job of caching. I tried different ways to "make test caches work" with little outcome. In a recent Go project, I cannot get the go build to use cached Go packages. So each time, a CI workflow runs, it downloads all packages like it has never seen them before. Other tools depends on build/test caches are also slow because of that. For example, golangci-lint takes 5 minutes each time it runs. On my Mac, it takes <30 seconds. I even think, CI SaaS providers intentionally make the cache hard because of they charge by "resource".

[aaronjensen]

If you still work where you worked a year or two ago, I'm actually responsible for introducing that company to and setting up bors. When a company has a monorepo and many developers attempting to get work merged in at the same time, bors can be used to ensure that those merges go sequentially and it even does added complicated things like merging multiple branches at once if it can to save "time".

However, if a company has separate repositories for separate things, the chances of people running into merge conflicts and deadlocks is extremely low and tools like bors become unnecessary. Lots of our tooling allow us to deal with underlying problems without actually correcting them. They are palliatives.

And yes, I hear you on caching. I can't say much good about it even when you get it working. Nothing was ever faster than when we had our own CI hardware and our "cache" was just the disks on those machines. We still use CI tooling for deployment out of necessity and have to rely on caching, which often takes longer to restore and persist than it would to install the dependencies from scratch. It's disturbing.

[ryanolson-aumni]

Some questions

Of note is that our newly authored JavaScript code does not use TypeScript, eslint, or prettier.

What does this mean for your development process? Do you ensure that your team members have prettier disabled for JavaScript files vs legacy TypeScript files?

If that sounds insane to you, then consider these two things: 1. Our team doesn't need these tools to thoughtfully adhere to our standards or be productive. As a matter of fact, we are more productive without them. 2. Our team employs specific countermeasures for specific circumstances

Do you know what sort of countermeasures? Does this mean digging more into having better tests? Or something else?

we catch them ourselves while we are attentively doing our work

I can see your point here. You can expect attentive and thoughtful development.

[aaronjensen]

Hi Ryan, good questions.

What does this mean for your development process? Do you ensure that your team members have prettier disabled for JavaScript files vs legacy TypeScript files?

We actually don't tend to touch the old TypeScript files much. That particular part of the project is functional, used, and successful. It doesn't have much reason to change, but at some point we will retrofit it as we add more capabilities (such that replacing it with equivalent functionality is relatively low effort). That said, we used prettier through ESLint, and our newer JavaScript code doesn't have an eslint config, so as long as people's editors are set up to only attempt to run ESLint if there is an ESLint config, they shouldn't get anything running, including prettier. If a person had their editor configured to run prettier, they would likely notice the first time they touched a JavaScript file and went to commit it. That should prompt them to correct their editor configuration and never commit it, so it wouldn't be a repeat problem.

We try and fix problems at their source, rather than continue to work around them.

That's important to note when you consider what ESLint, and Prettier are (TypeScript too, but it has other uses) -- they are breakfix tools. Their only value is detection, and detection is too late. Prettier is slightly different, but its problems are more severe -- it causes the developer to stop being concerned with formatting and thus lose touch with the fact that formatting is just one part of how they influence how their code is read and how easy that is. Brain on, not off.

"Quality comes not from inspection, but from improvement of the process" - Mary Walton
“You can not inspect quality into a product." — W. Edwards Deming

Do you know what sort of countermeasures? Does this mean digging more into having better tests? Or something else?

• Tests are certainly a countermeasure that obviates TypeScript
• Pairing or mobbing assist in attention
• We have a repository of norms, which are just text files describing certain ways that we do things. There aren't very many of them -- you could read through all of them over coffee.
• When mistakes are found, we discuss them at our daily sync as a team and reflect on countermeasures
• We encourage personal checklists
• We have specific checklists for certain things like verifying a new component before first deployment
• We encourage breaks/meditation/study/yoga/whatever helps keep you in focus
• We practice spotting mistakes during sync (a team member puts up a screenshot of code and the team takes a minute or two to write down on anonymous miro stickies what mistakes they see, and then we discuss)
• We work to ensure our entire codebase (all 400+ repos) adhere to our norms and standards so that everywhere we look is exemplary
• Etc.

I can see your point here. You can expect attentive and thoughtful development.

Indeed. Here's the reality: I've worked with some of the people on this team for 3-7 years in a row prior to starting this way of working. I've seen them grow more in the last 3 years by leaps and bounds than I did in the 3-7 prior. The entire team (this includes relatively jr devs, though i think our team members have anywhere from 5-30 years of experience, with many being around the 10 mark) brings energy, finds mistakes, corrects them, and gets better over time. At least those that stick around -- the work we do is intense, and it's not for everyone. It's not inhumane, at all. Quite the opposite. It's centered around the golden rule. We leave the code the way we would want to find it, we don't throw our fellow developers for a loop with oddly formatted, non-typical things.

See my article on special variation if you haven't: https://github.com/aaronjensen/software-development/blob/master/special-variation.md#special-cause-variation

Feel free to dig in further to anything you want here. I'm happy to discuss.

[doughtnerd]

That alert may seem helpful, but is it? What does it serve to prevent? Styling errors will not cause bugs for our users. They will likely be noticed the next time that a developer is in that codebase. We have successfully noticed when newcomers to the team do not use prettier to format their code and have instructed them to do so without need for automated checks.

I love this. I've felt this way for years. Though not always the case (especially on pet projects or very small libraries) I put a good deal of consideration into how things are formatted for readability's sake. I suppose readability is purely subjective but I often find formatting/styling from those tools make the code less readable.

Many of our packages have interactive tests as well. These are just as essential to validating our work, and they cannot be run by CI servers.

I'd be curious to know what these "interactive tests" are. My interpretation is that these are tests that would be well covered with some e2e tooling, no? I've worked in a couple places where qa had hundreds of "manual" tests. The DevOps handbook has a couple sections on such practices but suffice it to say, it's better to automate important tests like that. What aspect of these tests make it so CI can't run them? Perhaps I'm misunderstanding something.

When you push work that is validated by CI, there is some period of time before results are communicated back. A person must either wait patiently for the notification and/or monitor the run. More likely than that is that they continue work on something related, further encasing any mistakes that they are waiting for feedback on.

Definitely. I personally believe effective development is all about feedback cycles. The more quality feedback you can get and the shorter each of your cycles are, the better. For example, unit tests are a form of feedback cycle - they inform you of correctness with every change and are cheaper/quicker than CI. CI is another feedback cycle and deployment is another. Each feedback cycle combines to form your complete development flow. The shorter each cycle is and the less defects passed down to the next cycle, the better. If your first cycle (local development, running unit tests, compiling, etc) is fast and catches most/all of your defects, I really don't see a need for redundancy in CI. Doing so often lengthens the cycle and bottlenecks your process anyway.

Even with the advent of yaml based and re-usable CI configuration, this infrastructure must be maintained. It must be paid for both in a monthly service cost and in time spent configuring, tuning, and automating. We have 100s of repositories with tests, all of which would be subject to CI if that were part of our standard practice. That could be a significant amount of work to both set up and keep running over time.

I can definitely understand this. IMO, the maintenance cost is highly dependent on what tooling you choose (for example, I find GitHub actions to be incredibly low maintenance but Jenkins + self-hosted runners in K8's + Cloudbuild to be a lot higher) but I do understand that at scale, any tool can get hard to manage and very costly for questionable ROI.

[aaronjensen]

I suppose readability is purely subjective but I often find formatting/styling from those tools make the code less readable.

I don't think this is true. There may be subjective choices that can be made, but there are things we know about how most people take in information that leads us to some objectivity when we make our decisions. For example, we know that calling a function the argument place of another function requires more cognitive load than an explaining variable would in most cases.

I'd be curious to know what these "interactive tests" are.

It depends on the project. They are any test that benefit from having human eyes or hands on them. They may be semi-automated (autonomation -- automation with a human touch). We consider our UI interaction tests (e2e, as you referred to them) to be of this kind, though we do run them fully automated as well. We have a flag that slows them down and lets us observe them. We may run with that while developing the feature, but run them fully automated when testing a package update.

Other interactive tests may require a manual intervention in another system, or examining a message store after a series of messages is written.

it's better to automate important tests like that

Depends on the test, yes. We keep interactive tests to a minimum, but they are present where they need to be. Examining JavaScript interactive components is another case where the cost-benefit is often on the side of some interactive testing.

but I do understand that at scale, any tool can get hard to manage and very costly for questionable ROI.

At this point we have around 400 repos that would need CI if we had it. I'm glad we don't, because it means that there's zero friction from that when we need to introduce a new package.