topo support stale reads for consul

[demmer]

Description

Add optional support for AllowStale reads in the consul topo when polling for tablets in discovery.

Consul supports relaxed consistency on K/V reads as described in https://developer.hashicorp.com/consul/api-docs/features/consistency.

In particular there is a performant option for stale reads that allow lagged replies from a replica:

stale - Consul DNS queries use stale mode by default. This mode allows any server to handle the read regardless of whether it is the leader. The trade-off is very fast and scalable reads with a higher likelihood of stale values. Results are generally consistent to within 50 milliseconds of the leader, though there is no upper limit on this staleness. Since this mode allows reads without a leader, a cluster that is unavailable (no quorum) can still respond to queries.

None of the Vitess K/V options utilize this mode however, which means that all operations must go to the primary leader.

At the same time, the discovery TopologyWatcher as used by vtgate and other components repeatedly polls the tablet list to discover new tablets that have been added to the registry. As such, it can tolerate stale reads on a given request since it will turn around and re-request in the future.

To make this more efficient, this PR plumbs through an option so that this polling workloads sets the more relaxed AllowStale consistency option on the KV read, which should improve consul topo service performance.

Related Issue(s)

Checklist

• "Backport to:" labels have been added if this change should be back-ported
• Tests were added or are not required
• Did the new or modified tests pass consistently locally and on the CI
• Documentation was added or is not required

Deployment Notes