Enhancement: Configure reapers to distribute deletions evenly instead of in periodic bursts

[dynamic-entropy]

Enhancement Description

Sites with slow deletions struggle during high loads, often exceeding storage capacity and failing to catch up. We need to optimize reaper configuration and scheduling to address this issue. This enhancement should consider factors such as site-specific deletion latency and deletion backlog.

• In addition, auto-scaling and deploying reapers for specific sites during emergency high loads would be highly beneficial.

Use Case

During peak usage times, a site with slow deletions risks exceeding its storage capacity. Optimized reaper configurations and work scheduling will help manage deletions more efficiently.

• Automatic scaling and deployment of reapers will ensure that deletion processes keep pace with increased writes. This will also eliminate the manual workload for operators.

Possible Solution

No response

Related Issues

No response

[labkode]

Related to rucio/rucio#6512