Should we collapse drivers into a single repository?

[ssbarnea]

While I was the main driver behind the split of molecule drivers into separated repositories back in 2020, now I realised that this created a serious extra maintenance burden.

Should we bundle the relatively stable plugins into a repository like molecule-plugins or molecule-clouds and molecule-locals repos? Instead of having 15+ repos, having one or two might prove easier to cope with.

I even considering bringing them back into molecule itself but that would not be a very wise idea.

If we do the collapse we will have an extra for each plugin as we would not want to install all dependencies all the time as almost nobody needs all plugins at once.

If you ever helped with molecule plugin maintenance, please vote on the questions below. If you never did, you can skip it because the target is around people that do maintenance, as it is unlikely to affect the users much.

[ssbarnea]

@ansible-community/molecule please vote on this survey.

[cidrblock]

Single, but........ I think podman should be native/included to molecule

[apatard]

I've yet to make my mind on this. I understand the intent and I agree It's hard to follow the different repositories so everything in one place would make things easier (but not inside molecule git repository)

This also means that the plugins won't be handled separately so:

• if a quick release is needed for an important bug fix for one plugin, new bugs may be added since the others plugins may have been changed.
• Looking for changes in a plugin will be harder than just going to the plugin project page and look at the git log [I know, it's only few more clicks].
• How to know which plugin is affected by a bug when looking only at the page https://github.com/ansible-community/molecule-plugins/issues ? We'll have to create a tag by plugin ?
• in the case molecule-locals/molecule-clouds split, what would be the criteria to choose where a plugin goes ?

From a user point of view, installing all plugins in one or two commands will make things easier but not having the dependencies installed by default may surprise people. Unfortunately, installing all dependencies will be bad (bloat ?). As a side note, this makes me think that no matter what's decided, maybe we can create a "molecule full" package containing molecule and the plugins a little bit like ansible vs ansible-core ?

[ssbarnea]

Thanks for getting back on this. Basically we could add plugin dependencies as extras in order to allow installation of molecule-plugins package without dragging a huge amount of dependencies, especially as some cloud ones are very heavy like azure and aws. Still, due to packaging limitations the plugin entry points need to be registered by the package itself and they cannot depend on extra.

This means that the plugin will be installed and listed as installed even when its dependencies may be missing. That is not a really a huge problem as we can make them check their own dependencies at runtime, like we already do with collection dependencies.

The goal of combining the repos is to reduce maintenance on them related to CI/testing and keeping them compatible with molecule changes. If we count all changes that we had to do to cloud plugins, probably with have around 75% of them being counted as "chore" and unrelated to real bug-fixes. Combining them would lower that burden.

The idea of two bundles, a cloud based one and a local based one is related to usage patterns. There are also few other aspects:

• We are considering making podman plugin a core one, basically as this will likely be the only plugin that we might be able to package with molecule in a future version of AAP (like we did with ansible-lint in AAP 2.2). If podman goes into the bloat package, we might struggle to do it as we would not want to ship the entire kitchensink.
• If we combine all plugins into a single package, how are we going to build system packages for it? Basically can these extras translate to rpm/deb features that allow optionals? We need some feedback here as I do not want to rush into a direction that may prove to cause us a new set of problems.
• We need a proper survey of who is using molecule in production (testing pipelines) and with which plugins, getting that info is essential in order to realise where we need to invest time or not. For example. The fact that someone wrote a plugin 4 years ago, does not mean that it has at least one users this year, it could be as well as 100, or none. I would really want to avoid spending more effort trying keep plugins updated that maybe nobody is using, all that time could be spend on improving the more popular ones.

PS. It would be awesome to see comments from other molecule users, reporting how they use it with their CI/CD pipelines.

[amarao]

We've totally skipped all existing drivers in favor of our own delegated (a set of big playbooks to use Openstack in a way we like). The main problem we have is that code for 'prepare/destroy' is been copied more than 100 times already and it's really hard to to update if any bug found.

One thing which would make our life easier would be some kind of driver: delegated; source:[email protected]/our/driver:1.2.3 pointing to external repo (or pypi package, etc) to reuse that code.

[jseguillon]

Hi, I voted "one single repository". Here are my thoughts:

Pros: simplify user life, unify doc (maybe doc from drivers could be embedded on molecule readthedoc website ?), less complexity for CI.
Cons: real challenge on dependency, maintenance process and relases.

In the end, I believe improving developer experience worth the technical challenges. Hopefully we can find elegant solutions.

[jseguillon]

Maybe also we could do one repo for all, and publish two or more python packages ? We could even have one repo publishing full splitted packages like currently. This would ease CI and doc but would introduce no regression to actual behavior.