Highlight metrics exist for backup success #82
Conversation
| sum by (kubernetes_namespace) (increase(jobs_backup_resume_completed{kubernetes_namespace=~"auth|auth-customer"}[14h])) == 0 | ||
| ``` | ||
|
|
||
| While the backup job may have successfully run, that is not a full proof way of confirming you'll be able to restore it 😅 |
There was a problem hiding this comment.
out of curiosity: what are some of the stories here, is it possible to make a backup that's not actually usable?
There was a problem hiding this comment.
@matthewhughes-uw For the most part I'm airing on the side of caution, those stories of people backing up and then coming to restore to find it doesn't work.
That if you want full faith in the system you need to automate & check your restore process. Perhaps the message is alarmist here?
There was a problem hiding this comment.
Perhaps the message is alarmist here?
Not alarmist, I just found it raised more questions. I think your point might be value to document here
| While the backup job may have successfully run, that is not a full proof way of confirming you'll be able to restore it 😅 | |
| While the backup job may have successfully run, that is not a full proof way of confirming you'll be able to restore it. | |
| If you want full faith in the system you need to automate and check your restore process. |
There was a problem hiding this comment.
Question: where does this metric come from? I see on https://www.cockroachlabs.com/docs/stable/backup-and-restore-monitoring (maybe worth linking) there is a metric jobs.restore.resume_completed
EDIT: I looked at an older version of those docs https://www.cockroachlabs.com/docs/v22.2/backup-and-restore-monitoring and see it, so I guess those metrics were renamed?
|
I moved the PR here |
No description provided.