operator v1: store NodePoolSpec in STS annotations & refactor nodePool deletion slightly #323
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
birdayz
changed the title
birdayz
force-pushed
the
jb/virtual-nodepool-improvement
branch
from
368e073
to
c4b90ea
Compare
birdayz
changed the title
birdayz
requested review from
RafalKorepta,
chrisseto and
andrewstucki
as code owners
birdayz
changed the title
birdayz
force-pushed
the
jb/virtual-nodepool-improvement
branch
from
c4b90ea
to
fc0b719
Compare
birdayz
changed the title
birdayz
force-pushed
the
jb/virtual-nodepool-improvement
branch
from
fc0b719
to
de2e474
Compare
sbocinec
reviewed
Nov 21, 2024
birdayz
commented
Nov 21, 2024
| @@ -102,9 +102,8 @@ func (r *StatefulSetResource) handleScaling(ctx context.Context) error { | |||
| return r.setCurrentReplicas(ctx, *r.nodePool.Replicas, r.nodePool.Name, r.logger) | |||
| } | |||
|
|
|||
| if ptr.Deref(r.nodePool.Replicas, 0) == npCurrentReplicas && !r.nodePool.Deleted { | |||
| if ptr.Deref(r.nodePool.Replicas, 0) == npCurrentReplicas { | |||
There was a problem hiding this comment.
we do not need this special case anymore (great!)
RafalKorepta
approved these changes
Nov 21, 2024
| redpandaContainer = &container | ||
| break | ||
| var np vectorizedv1alpha1.NodePoolSpec | ||
| if nodePoolSpecJSON, ok := sts.Annotations[labels.NodePoolSpecKey]; ok { |
There was a problem hiding this comment.
Does this need to work with already existing NodePool specs? If there is no annotation here, what's the expected behavior?
There was a problem hiding this comment.
As part of the reconciliation, this is set. So once this version is deployed, it will update the StatefulSet and add the annotation.in addition, NodePools are not yet in production. The only case not covered is already in-deleting nodepools - but since there's no prod clusters yet, we'll be fine
There was a problem hiding this comment.
If we expect it to always be present, I feel we should handle the else case here either either an error or panic.
chrisseto
approved these changes
Nov 21, 2024
| redpandaContainer = &container | ||
| break | ||
| var np vectorizedv1alpha1.NodePoolSpec | ||
| if nodePoolSpecJSON, ok := sts.Annotations[labels.NodePoolSpecKey]; ok { |
There was a problem hiding this comment.
If we expect it to always be present, I feel we should handle the else case here either either an error or panic.
we encountered corner cases, where it becomes extremely difficult to synthesize a NodePoolSpec just by looking at the StatefulSet - which is our fallback, if a nodePool was removed from the spec. AdditionalCommandlineArguments is hard to reconstruct, because we'd need to pull out of of the args field in the pod spec of the STS, removing all "other default" args - very error prone. Instead, we now store the NodePoolSpec used to create the STS in the STS as an annotation. This way we can always find the NodePoolSpec to create the (deleted) STS. In addition, we take this chance to remove small special cases for handling delete nodepools: - Do not set replicas=currentReplicas anymore. It was more of a trick. Instead, we now set for a deleted nodePool replicas=0, which exactly represents what should happen with it (scale down to zero). - Add check for Deleted bool in scale-down handler. It prevented replicas=currentReplicas being accepted as "do notthing" if it's a deleted nodepool. Then, the control flow would proceed and downscaling happens. This was not very explicit and very hard to find out, why downscale even works in deleted NodePools. With the refactor, replicas is 0, and no special case is needed for deleting anymore.
birdayz
force-pushed
the
jb/virtual-nodepool-improvement
branch
from
de2e474
to
6921473
Compare
|
Good call out on the annotation-not-found case. added the else block. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
we encountered corner cases, where it becomes extremely difficult to synthesize a NodePoolSpec just by looking at the StatefulSet - which is our fallback, if a nodePool was removed from the spec. AdditionalCommandlineArguments is hard to reconstruct, because we'd need to pull that out of the args field in the pod spec of the STS, removing all "other default" args - very error prone.
in practice, this caused a rolling restart of a nodePool, because its arguments were not assembled correctly. in the moment it got deleted from spec, a diff came up (args missing).
To fix / solve correctly, we change our strategy. We now store the NodePoolSpec used to create the STS in the STS as an annotation. This way we can always find the NodePoolSpec to create the (deleted) STS.
In addition, we take this chance to remove small special cases for
handling delete nodepools:
Instead, we now set for a deleted nodePool replicas=0, which exactly
represents what should happen with it (intent to scale down to zero).
replicas=currentReplicas being accepted as "do notthing"
if it's a deleted nodepool. Then, the control flow would proceed and
downscaling happens. This was not very explicit and very hard to find
out, why downscale even works in deleted NodePools. With the refactor,
replicas is 0, and no special case is needed for deleting anymore.
This way, nodePool deletion works more like an ordinary scale down.