Meta: Suspicious replica recovery roadmap

[haozturk]

Feature Description

This is a meta issue to track down the steps to achieve suspicious replica recovery in CMS

• Short-term solution: Set up a cron job to check and recover suspicious replicas that CRAB reports
• Update WMArchive so that it provides PFNs to /topic/cms.jobmon.wmarchive
• Update rucio-tracers so that it propagates PFNs and error messages to Rucio
• Configure kronos with bad_file_patterns to mark replicas suspicious
• Fix the rucio problems that's preventing conveyor-finisher to mark replicas suspicious
• Deploy the suspicious_replica_recoveror daemon
• Deploy the minos_temporary_expiration daemon
• Implement monitoring

Use Case

https://indico.cern.ch/event/1356295/

Possible Solution

NA

Related Issues

This was the initial issue where this was discussed: #403
Rucio issues and PRs:

• Make suspicious-replica-recoveror daemon VO-agnostic rucio/rucio#6396
• Replicas: For declaring bad replicas, use plugin for deteriming scope rucio/rucio#6444