Leverage ApplicationSets for stacks/argo

[blairdrummond]

For your review @sylus , I think this is an alternative that might be nice. Basically the same idea but with base and overlays per app, and instead of an app-of-apps (which needs to manage the targetRevisions in a weird way), I think we could plop a single ApplicationSet in the aaw-argocd-manifests repo.

Let me know what you think!

CC @wg102 @rohank07

Edit: Still in draft, testing now

[sylus]

This is good but we should keep the concept of stacks since that is the naming Kubeflow uses, and ArgoCD should only be one of the deployment methodologies and not override the default manifest generation.

When I want to quickly test Kubeflow I want to just install their manifests directly with our customizations and not have to deal with Argo or add anything more to my mental model other then I'm calling their manifests at a pinned version with kustomize adjustments and I am rendering the yaml.

For example its extremely useful to just run to get what the full YAML looks like:

kubectl kustomize stacks/aaw > output/

Obviously we will be using Argo ApplicationSets pattern but that should stay in stacks/argo. From a testing perspective etc its beneficial we just treat ArgoCD and project metadata around the manifests as separate and not intertwined.

Also for mental model is very beneficial from a birds eye view to see all the components of what will be installed for instance under AAW stack (we will make notes in this file about what comes from CN platform). This is also exactly how Kubeflow constructs theres with comments and separation.

• https://github.com/kubeflow/manifests/search?q=stacks
• https://github.com/kubeflow/manifests/blob/master/example/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:

## Kubeflow (common)

- ../../common/kubeflow-namespace/
- ../../common/kubeflow-roles/
- ../../common/oidc-authservice/

## Kubeflow (apps)

- ../../apps/katib
- ../../apps/kfserving
- ../../apps/pipeline

[sylus]

Also I think @wg102 wanted to go v1.3.1 and not jump straight to v1.4.1 so I made that change and pushed to a new main branch which can be worked from.

[zachomedia]

I'm going to jump in with a couple of recommendations from experience.. it's best, if possible, to separate the "artifact" from the deployment methodology. This comes with a few benefits:

• Easy to test in various scenarios, down to even just rendering the manifests
• Deployment tools can change
• Separation from artifact and configuration

Therefore, what @sylus wrote is good advice and I second that recommendation.

I would say this follows the same model as other deployment tools:

• terraform: module vs. configuration
• helm: chart vs values

It's best practice to decouple to two from each other.

[blairdrummond]

Yeah these ideas make sense, I think the main thing I want in the design is to ensure that each component and its variations can be tested independently. Can you illustrate how you would create an aaw-related kustomization in this design, and how you'd keep the argo and aaw stack in sync with that added change? Maybe it's just hard to see for me atm because the examples all point to the same spot.

[sylus]

I took the large bulk of this and attributed to you :) i just kept the stack naming and didn't make use of overlays yet since the point at the moment is to just capture aaw state.

• 4cccd53

Argo is really just the project metadata for aaw stack and local and upstream won't need argo instances.

I pointed @wg102 and @rohank07 to #116 as a guide ^_^

[blairdrummond]

Oh ok, if all kustomizations (in the form of overlays) happens in the apps/common/contrib folder, and the stacks just aggregate those resources+versions, then that makes a lot of sense to me!

Only thing is I'm not 100% sure whether the ApplicationSet should actually live in this repo v.s. the aaw-argocd-manifests repo. I could see this repo holding the artifacts, but then the dev/prod branches/kustomizations in the aaw-argocd-manifests repo manages the tags of the artifacts being deployed.

Does that kinda make sense to you @sylus ?

[sylus]

Yeah i was thinking u would have it in yours, though i might keep ours just as an example for users.

I think I addressed most of what was in this P.R. so going to close this one out.