Make Remote Publication a dynamic setting

[shiv0408]

Description

Make the cluster.remote_store.publication.enabled a Dynamic setting. Also, made clear distinction of isPublicationEnabled methods and isPublicationConfigured methods. We should only use the isPublicationEnabled methods on active ClusterManager node, otherwise is can give inconsistent results.

Related Issues

Resolves #15852

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 68c358c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for a15a1b6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for c2b26a2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for cb46759: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 91d4e60: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 72.00000% with 7 lines in your changes missing coverage. Please review.

Project coverage is 71.95%. Comparing base (20536ee) to head (4e9148b).
Report is 25 commits behind head on main.

Files with missing lines	Patch %	Lines
...arch/gateway/remote/RemoteClusterStateService.java	66.66%	1 Missing and 5 partials ⚠️
...ting/remote/InternalRemoteRoutingTableService.java	50.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #15937      +/-   ##
============================================
- Coverage     72.00%   71.95%   -0.06%     
+ Complexity    64796    64794       -2     
============================================
  Files          5307     5307              
  Lines        302540   302553      +13     
  Branches      43708    43705       -3     
============================================
- Hits         217853   217702     -151     
- Misses        66801    67009     +208     
+ Partials      17886    17842      -44     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Bukhtawar]

Sorry what is the behaviour on disabling a remote publication? Will it perform a full cluster state publication over transport

[github-actions]

✅ Gradle check result for 4e9148b: SUCCESS

[soosinha]

Sorry what is the behaviour on disabling a remote publication?

@Bukhtawar The cluster manager node always tries to send the diff cluster state over transport layer. If the target node does not have previous version then the full state is sent. The same is applicable when remote publication is disabled

[Bukhtawar]

Sorry what is the behaviour on disabling a remote publication?

@Bukhtawar The cluster manager node always tries to send the diff cluster state over transport layer. If the target node does not have previous version then the full state is sent. The same is applicable when remote publication is disabled

Understand, my point is if we switch from remote to local transport publication, should we reconcile the state first in case remote publication had some discrepancy applying the state and probably also the reason why the remote publication also needs to be disabled in the first place

[soosinha]

@Bukhtawar If remote publication has some issue, then the checksum validation should prevent such publication.

[soosinha]

Created an issue to track publication of full cluster state on disabling remote publication: #16345

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-15937-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0c3e3c009ae85943460bcc26805c1cf57b195b91
# Push it to GitHub
git push --set-upstream origin backport/backport-15937-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-15937-to-2.x.