Leader Election Only Cluster with Random Startup Order

[martinzwirner]

Hi,

I'm using DotNext.AspNetCore.Cluster to implement leader election in a cluster of three nodes with one standby node and TCP transport. I have two questions about the proper set up:

1. I understand that in this case the default ConsensusOnlyState implementation can be used and that it does not store state on disk. IMHO this is no problem as the nodes are auto-discovered and added to the cluster on startup. Should a persistent storage be configured anyway and if yes, how?

2. I understand that ColdStart = true should be set only on the node that's started first. In my case users should not have to do this manually though. My current solution is to set ColdStart = true on both non-standby nodes. However this seems to create errors sometimes when both nodes are started at the same time. Is my approach correct or is there a better way?

Thank you!

[sakno]

Should a persistent storage be configured anyway and if yes, how?

Yes, you can use RaftClusterConfiguration.UsePersistentConfigurationStorage extension method for that.

Is my approach correct or is there a better way?

No, bootstrapping requires a single node with ColdStart=true if cluster configuration is empty. Alternative approach is to populate cluster configuration with some members programmatically, the configuration should be the same for all nodes at the time of bootstrap. Otherwise, the node don't know what to do. There are two possible scenarios:

1. You trying to bootstrap a new cluster from scratch. That's why you need a cluster with single node and this node should be a leader. This is ColdStart=true.
2. You trying to add a new member to existing cluster. In this case the node waits for catch up procedure executed by the existing leader. The node needs to be added to existing cluster configuration of the leader (announced). This is ColdStart=false.

[martinzwirner]

Thank you for your response @sakno.

RaftClusterConfiguration.UsePersistentConfigurationStorage is not working for me since it registers a service of type IClusterConfigurationStorage<UriEndPoint>. I'm using TCP transport though and need IClusterConfigurationStorage<EndPoint>. Did I miss something?

[sakno]

I see.. Yes, DI will not help you with that, I thought you using AspNetCore extensions for Raft. RaftCluster.TcpConfiguration.ConfigurationStorage property is what you're looking for. You can derive from PersistentClusterConfigurationStorage<TAddress> with EndPoint generic argument. EndPointFormatter will help you to implement necessary abstract methods.

[martinzwirner]

I still have a problem setting ColdStart. If I understood correctly, this should be set to true on exactly one node, right? But I don't know in advance which node will be started first (i.e. if its a new cluster or not). So I see two options:

• User needs to set ColdStart manually. This I want to avoid.
• I need to detect if another node is already started and then set ColdStart accordingly. I don't see how I can do this without creating a race condition.

You wrote:

Alternative approach is to populate cluster configuration with some members programmatically

Can you show me some code how to populate the cluster configuration with some members programmatically?

the configuration should be the same for all nodes at the time of bootstrap

I'm using autodiscovery to discover the other nodes. I don't think I can guarantee this.

Thank you for your help!

[sakno]

If I understood correctly, this should be set to true on exactly one node, right?

Right. If this approach cannot be provided, bootstrap all nodes with the same configuration.

Can you show me some code how to populate the cluster configuration with some members programmatically?

See my previous message about that.