Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Swap to a FIFO queue #1147

Draft
wants to merge 1 commit into
base: develop
Choose a base branch
from

Conversation

BryanFauble
Copy link
Contributor

Problem:

  1. AsyncIO has no order guarentee for the AsyncIO tasks that are created, it can lead to us requesting a auth token for a specific file and not using it until much later, leading it to be expired.

Solution:

  1. Try out moving to a FIFO queue for file downloads that works by indexing the container to be downloaded, and then starting a number of workers that will concurrently download each file. This makes it such that only a smaller number of files can enter the download process rather than everything.

Testing:

  1. More testing is needed

@github-advanced-security
Copy link

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant