Use TSCs in-place of Pod affinity/anti-affinity & always enforce them even if custom placement is specified

[malayparida2000]

Currently TSCs are enforced only if there is no other placement spec defined. Without the TSCs, the pods might not be distributed evenly. So we should always enforce the TSCs even if some custom placement is specified.

This also refactors the code to use TSCs in-place of Pod affinity/anti-affinity for ceph daemons like mgr, mon, mds, rgw, nfs.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: malayparida2000
Once this PR has been reviewed and has the lgtm label, please assign nbalacha for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[travisn]

This also refactors the code to use TSCs in-place of Pod affinity/anti-affinity for ceph daemons like mgr, mon, mds, rgw, nfs.

While I would expect the TSCs to work in place of the pod anti-affinity, it also feels like it has a bit of risk. Could you confirm some manual testing with this before merge? And we will want to point out to QE to watch for placement issues in case of regression.

Alternatively, it seems lower risk to always append our pod anti-affinity to the user's placement, instead of replacing it with a TSC.

[malayparida2000]

This also refactors the code to use TSCs in-place of Pod affinity/anti-affinity for ceph daemons like mgr, mon, mds, rgw, nfs.

While I would expect the TSCs to work in place of the pod anti-affinity, it also feels like it has a bit of risk. Could you confirm some manual testing with this before merge? And we will want to point out to QE to watch for placement issues in case of regression.

Alternatively, it seems lower risk to always append our pod anti-affinity to the user's placement, instead of replacing it with a TSC.

Considering the high risk nature of this I am thinking of not taking this to 4.17 but keeping it in 4.18 only but nothing concrete yet. I think TSCs satisfy our requirement of even distribution perfectly & they are a cleaner approach then pod anti-affinity. So weighing on using TSCs always.

[malayparida2000]

/hold