[Doc] Best practice to setup Redis for GCS FT

[kevin85421]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Create a user guide for configuring a RayCluster with GCS Fault Tolerance using Redis on AWS or GCP. The guide should include persistence option for Redis to ensure Redis state can be recovered after a restart.

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[andrewsykim]

Options for Redis persistence https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/

[andrewsykim]

I put together this gist to persist Redis using GCSFuse https://gist.github.com/andrewsykim/55088178684b5a692854f932c8120914

[andrewsykim]

@rueian Kai-Hsun mentioned you're a Redis expert, do you have opinions on whether we should use RDB or AOF for persistence?

[rueian]

Hi @andrewsykim,

Generally, AOF is better since it persists changes more frequently in the format of append logs while RDB is the periodic snapshot of the whole redis memory.

However, when it comes to the integration of GCSFuse, I believe RDB is better because GCS doesn't support append operations. If we use AOF with GCSFuse, it will need to re-upload the whole aof file again and again whenever there is a new entry appended and the aof file will get bigger and bigger and then slows down redis in this case.

[andrewsykim]

If we use AOF with GCSFuse, it will need to re-upload the whole aof file again and again whenever there is a new entry appended and the aof file will get bigger and bigger and then slows down redis in this case.

Is this a problem specific to GCSFuse? Wouldn't this be a problem for any file-system based approach?

[rueian]

It is not specific to GCSFuse. Most cloud storages doesn’t support append operations, except for Azure blob storage and AWS S3 Express One Zone.

AOF getting bigger will not be a problem for other file systems which support append operations because they don’t need to rewrite the whole file when appending a new entry.

[andrewsykim]

okay makes sense, so we should either use block storage for AOF or RBD persistence if using GCSFuse

[spencer-p]

I'm interested in getting this doc together

[spencer-p]

Without clustering redis (all of the current guides have 1 replica), we're talking about using Redis as a save-to-disk engine. I'd like to include some clustering and failover to our best practices recommendations to get the most out of redis.

For reentrant workloads that can handle rolling back a few minutes, I wonder if it would be simpler/cheaper to write GCS to disk occasionally.

[rueian]

we're talking about using Redis as a save-to-disk engine.

Exactly.

I'd like to include some clustering and failover to our best practices recommendations to get the most out of redis.

Ray only supports standalone Redis (single master). It doesn’t support Redis Cluster (multiple sharded masters) and Redis Sentinel, the two types of clustering have failover built-in. So, now users needs to implement Redis HA by themselves.

As far as I know, https://github.com/dragonflydb/dragonfly-operator is the only open-source solution that has automatic failover.

For reentrant workloads that can handle rolling back a few minutes, I wonder if it would be simpler/cheaper to write GCS to disk occasionally.

Do you mean you want to skip Redis?