DotNext.Net.Cluster.Consensus.Raft: Seed members and joining members?

[rogeralsing]

Is the DotNext.Net.Cluster.Consensus.Raft cluster capable of allowing non-specified members to join the cluster?
e.g. cluster boots with 3 seed nodes, then additional nodes join in against one of those, and the cluster become aware of the newly joined nodes?

Or is it purely capable of consensus between the pre-specified members?

In https://github.com/AsynkronIT/protoactor-dotnet , we have externalized all cluster management.
e.g. we use Consul or Kubernetes API to get the cluster topology.

I would very much like to give it a try to use dotNext Raft as an alternative cluster provider, as it wouldn't require users to have some specific infrastructure in place.

[sakno]

Hi @rogeralsing ,
Short answer is yes but it requires service discovery.

Long answer:
The current implementation need to know about the list of cluster members. It's needed for calculating quorum. Under the hood, the configuration of cluster members can be supplied via ASP.NET Core configuration infrastructure (in case of HTTP transport) or manually (in case of TCP/UDP transport). Configuration providers in ASP.NET Core can be extended. This gives a way to write custom provider and discover cluster nodes using 3rd-party mechanism such as Consul or Kubernetes. However, cluster members can be added and removed dynamically to live cluster. Raft implementation for ASP.NET Core (which is in DotNext.AspNetCore.Cluster library) tracks membership changes in the background using IOptionsMonitor. It's fully tested functionality. Moreover, there is a special interface for that: IExpandableCluster. If cluster engine implements it, the cluster can be re-configured dynamically. It's implemented for ASP.NET Core out-of-the-box but not for TCP/UDP transports.

Writing custom configuration providers for the purpose of service discovery may be challenging and redundant. Therefore, in version 3.1.0 (which is in development now) a new way of service discovery is introduced: IMemberDiscoveryService interface. It's much simpler than custom configuration provider. It allows to implement any custom discovery mechanism. Also I tried to write my own discovery mechanism for Kubernetes which is based on Service abstraction to verify the concept.

This interface is introduced because there are too many hosting infrastructures exist: Kubernetes, clouds, pure Docker, bare metal etc. As a result, I can't offer universal implementation. Instead, it's better to provide extensibility point to the developers.

In other words, service discovery is delegated to library consumers. From my point of view, it's not a goal of Raft implementation to provide built-in universal discovery mechanism. In case of bare metal deployment, some solutions such as Hazelcast IMDG use their own discovery mechanism based on UDP broadcast. .NET Framework has PRNP but it looks like dead (Windows-only, not ported to .NET Core). UDP broadcasting has many drawbacks and network engineers usually don't like this approach.

I think you're looking for some default service discovery mechanism. Raft is about building consistent and reliable clusters.

[sakno]

@rogeralsing , Raft-native membership management has been added to 4.0.0 version. It's available as beta version for now. There is no need to use external discovery mechanism anymore. The cluster is capable to manage membership changes as described in the original Raft paper.

[sakno]

@rogeralsing , I would like to inform you about upcoming changes in the next minor versions of DotNext.Net.Cluster and DotNext.AspNetCore.Cluster libraries. They were introduced in .NEXT as tools for writing clustered microservices. At the moment, only Raft consensus algorithm is implemented as a part of this suite. But now I'm working on extending the framework with additional features:

• Gossip-based communication
  • HyParView membership protocol for peer discovery with thousands of nodes
  • SWIM membership protocol for consistent sync of members in clusters based on Raft and Paxos.
• JumpHash as a basic building block for sharding

HyParView and SWIM will be provided as transport-agnostic and HTTP implementations. Depends on your needs, you will be able to utilize one of these implementations for discovering peers in the cluster.

[aley1]

@sakno I have found this old thread and it relates to the problem I have facing at the moment. I am trying to get DotNext Tcp Raft to work with a dynamic membership architecture.
When member nodes are known in advance, and added in the Configuration during startup (e.g. 5 members each started with a config file that provides the list of members and endpoints) - everything works well.
However I am attempting to do this:

1. Start 1st node in coldstart mode (no knowledge of other members) - this will become a Leader node
2. Start a 2nd node which sends request to 1st node to join cluster. 1st Node executes AddMembersAsync - this works, node joins cluster. membership count=2
3. Start 3rd node, 4th, 5th nodes using step 2. membership count = 5
4. Leader is 1st Node
5. Kill 1st Node - expecting other nodes to take over as Leader
6. What I see if Leader is still shown as 1st Node (after 1st node is already offline) - no election takes place

I've also tried step 1. without starting as coldstart node but adding members does not work.

Could you please advise what am I doing wrong here?

[aley1]

Also wanted to add that after sending request to 1st Node, I upgraded the joining nodes from standby to normal via RevertToNormalModeAsync();

[sakno]

It\s better to create a new issue.