Availability set deprecation #1025

kon-angelo · 2024-11-29T08:42:38Z

How to categorize this PR?

/area control-plane
/kind enhancement
/platform azure

What this PR does / why we need it:

Deprecate Availability-Set based shoots
Make VMSS-FLex (VMO) the default for non-zonal clusters
Create a migration path based on https://learn.microsoft.com/en-us/azure/virtual-network/ip-services/public-ip-basic-upgrade-guidance to migrate AS to VMO shoots.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Release note:

Disable the creation of Availability-Set-based shoots.

VMSS-Flex based shoots are not the default deployment for non-zonal shoots.

Introduce an annotation to migrate the availability-set shoots to VMSS-Flex shoots.

kon-angelo · 2024-11-29T08:42:44Z

/test

testmachinery · 2024-11-29T08:42:48Z

Testrun: e2e-xn69g
Workflow: e2e-xn69g-wf
Phase: Succeeded

+---------------------+---------------------------+-----------+----------+
|        NAME         |           STEP            |   PHASE   | DURATION |
+---------------------+---------------------------+-----------+----------+
| infrastructure-test | infra-flow-test           | Succeeded | 21m14s   |
| infrastructure-test | infra-flow-migration-test | Succeeded | 26m21s   |
| bastion-test        | bastion-test              | Succeeded | 9m4s     |
| infrastructure-test | infrastructure-test       | Succeeded | 23m8s    |
+---------------------+---------------------------+-----------+----------+

AndreasBurger · 2024-11-29T09:25:55Z

pkg/controller/infrastructure/infraflow/ensurer.go

+		if err := fctx.client.Get(ctx, k8sclient.ObjectKey{
+			Namespace: fctx.infra.Namespace,
+			Name:      azure.CloudControllerManagerName,
+		}, deployment); k8sclient.IgnoreNotFound(err) != nil {
+			return err
+		} else if k8sclient.IgnoreNotFound(err) != nil {
+			return nil
+		}


Suggested change

if err := fctx.client.Get(ctx, k8sclient.ObjectKey{

Namespace: fctx.infra.Namespace,

Name: azure.CloudControllerManagerName,

}, deployment); k8sclient.IgnoreNotFound(err) != nil {

return err

} else if k8sclient.IgnoreNotFound(err) != nil {

return nil

}

if err := fctx.client.Get(ctx, k8sclient.ObjectKey{

Namespace: fctx.infra.Namespace,

Name: azure.CloudControllerManagerName,

}, deployment); k8sclient.IgnoreNotFound(err) != nil {

return err

}

Not sure if this is the right one to keep

pkg/controller/infrastructure/infraflow/flow_context.go

pkg/azure/client/loadbalancers.go

hebelsan · 2024-11-29T11:15:59Z

pkg/controller/infrastructure/infraflow/ensurer.go

@@ -171,7 +177,16 @@ func (fctx *FlowContext) EnsureAvailabilitySet(ctx context.Context) error {
 		return nil
 	}

-	avset, err := fctx.ensureAvailabilitySet(ctx, log, *avsetCfg)
+	// complete AS migration.
+	if v := fctx.whiteboard.GetChild("migration").GetChild(KindAvailabilitySet.String()).Get("complete"); v != nil && *v == "true" {


Nit: I think it would be nice we define such keys as "migration" and "complete" somewhere as constants.

hebelsan · 2024-11-29T13:21:29Z

pkg/controller/infrastructure/infraflow/ensurer.go

+	// return early if the migration has already been complete
+	if v := fctx.whiteboard.GetChild("migration").GetChild(KindAvailabilitySet.String()).Get("complete"); v != nil && *v == "true" {
+		return nil
+	}
+	// return early if the cluster does not have AS.
+	if fctx.whiteboard.GetChild(ChildKeyIDs).Get(KindAvailabilitySet.String()) == nil {
+		return nil
+	}
+
+	// IF VMOs are not needed, or the migration is already done, return early.
+	if !helper.HasShootVmoMigrationAnnotation(fctx.cluster.Shoot.GetAnnotations()) {
+		return nil
+	}


Nit: Aren't these all dependencies for the migration task - maybe we could put them into a separate function and use this function as task dependency.

These checks as a group are used in one place. The separate function will just be giving a name to preflight checks but I can oblige

Rethinking your question:

Do you mean as a shared.DoIf ?
It's a question of when are the dependencies of a task evaluated (at build or at runtime) and AFAIK they are on build time. At that point you may not have the KindAvailabilitySet in the state for example.

Do you mean as a shared.Dependency ?
I am actually not sure if this will block a child Task in the graph from being executed. I would need to test the scemantics. It may either mark the skipped parent as completed (so the childen are always executed), or it may really prune the tree (what you suggested I believe). But I need to check first.

Based on https://github.com/gardener/gardener/blob/84184385dbb9616c2c838e4f24487cae2be75da0/pkg/utils/flow/flow.go#L303 I think it marks the skipped nodes as completed

Valid arguments.
I think I was a bit too quick here and didn't think it through completely.
Let's leave it as is.

hebelsan · 2024-11-29T13:48:16Z

pkg/controller/infrastructure/infraflow/infra_adapter.go

+}
+
+// IsVmoRequiredForInfrastructure determines if VMO is required.
+func (ia *InfrastructureAdapter) IsVmoRequiredForInfrastructure() bool {


I don't see where this function is used?

old artifact - thank you

hebelsan · 2024-11-29T14:06:26Z

pkg/controller/infrastructure/infraflow/migrations.go

+	if lb.Properties == nil || len(lb.Properties.FrontendIPConfigurations) == 0 {
+		return nil
+	}


duplicated check

AndreasBurger · 2024-12-02T12:30:22Z

pkg/controller/infrastructure/infraflow/ensurer.go

+	}
+	log.Info("Deleting load balancer", "Name", fctx.infra.Namespace)
+	if err := loadbalancerClient.Delete(ctx, fctx.adapter.ResourceGroupName(), fctx.adapter.TechnicalName()); err != nil {
+		return err
+	}
+	log.Info("Deleting internal load balancer", "Name", fctx.infra.Namespace)
+	if err := loadbalancerClient.Delete(ctx, fctx.adapter.ResourceGroupName(), fmt.Sprintf("%s-internal", fctx.adapter.TechnicalName())); err != nil {
+		return err
+	}


Suggested change

}

log.Info("Deleting load balancer", "Name", fctx.infra.Namespace)

if err := loadbalancerClient.Delete(ctx, fctx.adapter.ResourceGroupName(), fctx.adapter.TechnicalName()); err != nil {

return err

}

log.Info("Deleting internal load balancer", "Name", fctx.infra.Namespace)

if err := loadbalancerClient.Delete(ctx, fctx.adapter.ResourceGroupName(), fmt.Sprintf("%s-internal", fctx.adapter.TechnicalName())); err != nil {

return err

}

}

loadbalancerName := fctx.adapter.TechnicalName()

log.Info("Deleting load balancer", "Name", loadbalancerName)

if err := loadbalancerClient.Delete(ctx, fctx.adapter.ResourceGroupName(), loadbalancerName); err != nil {

return err

}

loadbalancerName = fmt.Sprintf("%s-internal", fctx.adapter.TechnicalName())

log.Info("Deleting internal load balancer", "Name", loadbalancerName )

if err := loadbalancerClient.Delete(ctx, fctx.adapter.ResourceGroupName(), loadbalancerName ); err != nil {

return err

}

AndreasBurger · 2024-12-02T12:31:19Z

pkg/controller/infrastructure/infraflow/ensurer.go

+		deployment.Spec.Replicas = ptr.To(int32(0))
+		// Apply the patch on the "scale" subresource
+		if err := c.SubResource("scale").Patch(ctx, deployment, patch); err != nil {
+			return fmt.Errorf("failed to patch deployment %s with replicas: %w", key.String(), err)


Suggested change

return fmt.Errorf("failed to patch deployment %s with replicas: %w", key.String(), err)

return fmt.Errorf("failed to patch deployment %s with replicas: %w", key, err)

should not be needed

AndreasBurger · 2024-12-02T12:46:19Z

pkg/controller/infrastructure/infraflow/infra_adapter.go

+	if ia.config.Zoned {
+		return false
+	}
+	// If the infrastructureStatus already exists that mean the Infrastucture is already created.


Suggested change

// If the infrastructureStatus already exists that mean the Infrastucture is already created.

// If the infrastructureStatus already exists that means the Infrastucture is already created.

kon-angelo · 2024-12-03T15:38:24Z

/test

testmachinery · 2024-12-03T15:38:30Z

Testrun: e2e-f5mbk
Workflow: e2e-f5mbk-wf
Phase: Succeeded

+---------------------+---------------------------+-----------+----------+
|        NAME         |           STEP            |   PHASE   | DURATION |
+---------------------+---------------------------+-----------+----------+
| infrastructure-test | infrastructure-test       | Succeeded | 22m44s   |
| infrastructure-test | infra-flow-test           | Succeeded | 20m5s    |
| infrastructure-test | infra-flow-migration-test | Succeeded | 25m4s    |
| bastion-test        | bastion-test              | Succeeded | 9m30s    |
+---------------------+---------------------------+-----------+----------+

implement migration

039256c

kon-angelo requested review from a team as code owners November 29, 2024 08:42

gardener-robot added the needs/review Needs review label Nov 29, 2024

kon-angelo changed the title ~~Availability set depraction~~ Availability set deprecation Nov 29, 2024

AndreasBurger reviewed Nov 29, 2024

View reviewed changes

gardener-robot-ci-1 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Nov 29, 2024

fix ccm not found error handling

ba7a8a1

kon-angelo force-pushed the avset-deprecation3 branch from dbc9ae3 to ba7a8a1 Compare November 29, 2024 09:47

gardener-robot-ci-2 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Nov 29, 2024

hebelsan reviewed Nov 29, 2024

View reviewed changes

redundant checks and consts for strings

4b3f021

gardener-robot-ci-1 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Nov 29, 2024

AndreasBurger previously approved these changes Dec 2, 2024

View reviewed changes

cosmetics

c0a9e15

kon-angelo dismissed AndreasBurger’s stale review via c0a9e15 December 2, 2024 16:41

gardener-robot-ci-2 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Dec 2, 2024

AndreasBurger approved these changes Dec 3, 2024

View reviewed changes

kon-angelo merged commit 3f41c47 into gardener:master Dec 3, 2024
11 checks passed

kon-angelo deleted the avset-deprecation3 branch December 3, 2024 16:05

gardener-robot added the status/closed Issue is closed (either delivered or triaged) label Dec 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Availability set deprecation #1025

Availability set deprecation #1025

kon-angelo commented Nov 29, 2024

kon-angelo commented Nov 29, 2024

testmachinery bot commented Nov 29, 2024 •

edited

Loading

AndreasBurger Nov 29, 2024

hebelsan Nov 29, 2024

hebelsan Nov 29, 2024

kon-angelo Nov 29, 2024

kon-angelo Nov 29, 2024 •

edited

Loading

kon-angelo Nov 29, 2024

hebelsan Dec 1, 2024

hebelsan Nov 29, 2024

kon-angelo Nov 29, 2024

hebelsan Nov 29, 2024

AndreasBurger Dec 2, 2024

AndreasBurger Dec 2, 2024

AndreasBurger Dec 2, 2024

kon-angelo commented Dec 3, 2024

testmachinery bot commented Dec 3, 2024 •

edited

Loading

	return fmt.Errorf("failed to patch deployment %s with replicas: %w", key.String(), err)
	return fmt.Errorf("failed to patch deployment %s with replicas: %w", key, err)

	// If the infrastructureStatus already exists that mean the Infrastucture is already created.
	// If the infrastructureStatus already exists that means the Infrastucture is already created.

Availability set deprecation #1025

Availability set deprecation #1025

Conversation

kon-angelo commented Nov 29, 2024

kon-angelo commented Nov 29, 2024

testmachinery bot commented Nov 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kon-angelo Nov 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kon-angelo commented Dec 3, 2024

testmachinery bot commented Dec 3, 2024 • edited Loading

testmachinery bot commented Nov 29, 2024 •

edited

Loading

kon-angelo Nov 29, 2024 •

edited

Loading

testmachinery bot commented Dec 3, 2024 •

edited

Loading