Avoid reprocessing data in backfill-index

[JasonPowr]

Issue: #2279

Summary

Motivation

This pr helps to solve the issue of reprocessing already indexed data during repeated runs of the backfill-index job by adding a --enable-redis-index-resume flag to backfill-index. When enabled, backfill-index saves the last processed index in Redis and resumes from it in subsequent runs, processing only new entries.

Release Note

Added a --enable-redis-index-resume flag to backfill-index, enabling jobs to resume from the last processed index and process only new entries for improved efficiency.

Documentation

[haydentherapper]

Can you use --start?

[JasonPowr]

Can you use --start?

Hi @haydentherapper, normally yes, however we tend to run the backfill-index on a schedule e.g a cronjob and keeping '--start' update to date in this context tends to be awkward, we either have to manually do it or store the value for '--start' in an awkward way which could be overwritten or modified.

[JasonPowr]

@haydentherapper When you get a chance can I get a review on this :), thanks

[haydentherapper]

What is the use case for running this periodically? It’s expected to be run exceptionally, when there is an outage that’s caused a loss of index entries.

How would you handle —end? If this is a cronjob, are you always processing up the last entry?

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 44 lines in your changes missing coverage. Please review.

Project coverage is 49.80%. Comparing base (488eb97) to head (55041c2).
Report is 250 commits behind head on main.

Files with missing lines	Patch %	Lines
cmd/backfill-index/main.go	0.00%	44 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #2280       +/-   ##
===========================================
- Coverage   66.46%   49.80%   -16.66%     
===========================================
  Files          92      192      +100     
  Lines        9258    24725    +15467     
===========================================
+ Hits         6153    12314     +6161     
- Misses       2359    11320     +8961     
- Partials      746     1091      +345     

Flag	Coverage Δ
e2etests	46.49% <ø> (-1.06%)	⬇️
unittests	41.83% <0.00%> (-5.85%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[JasonPowr]

What is the use case for running this periodically?

It was recommend to us to run the cronjob periodically, when we were first getting started with sigsore, as if there are errors writting to the index for any reason, we can make sure that it always manages to stay up to date.

How would you handle —end? If this is a cronjob, are you always processing up the last entry?
Essentially we are curling the rekor api and using the treeSize as the --end value

Hope that helps to clear things up :)

[haydentherapper]

It was recommend to us to run the cronjob periodically

This shouldn't be needed. Failures would be exceptional, e.g. if Redis goes down, and they can be monitored for in the logs as well.

Technically, I don't have an issue with this PR, I just don't think it's necessary. I'm happy to merge this if you still need to be reprocessing data.

[JasonPowr]

Apologies @haydentherapper linter was giving me trouble :)

Technically, I don't have an issue with this PR, I just don't think it's necessary. I'm happy to merge this if you still need to be reprocessing data.

Thank you that would be great :) and thanks for you input also :)

[cmurphy]

This is setting the last filled index to the --end index, which is already known to the script caller since it must be provided. Why not just have the cron job set this value itself when the script completes, and retrieve it to provide to --start when calling the script again?

[JasonPowr]

Why not just have the cron job set this value itself when the script completes, and retrieve it to provide to --start when calling the script again?

@cmurphy So something like that is definitely possible, however I feel like storing it in this manner is much more reliable, it also makes it more difficult to modify if someone wanted to, given that it could be locked behind a auth, as opposed to written to some env var or file somewhere

[cmurphy]

It could just write to the redis instance itself? With redis-cli and the auth credentials you're already passing to the script.