Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve upload performance #3

Open
mzur opened this issue Apr 7, 2022 · 1 comment
Open

Improve upload performance #3

mzur opened this issue Apr 7, 2022 · 1 comment

Comments

@mzur
Copy link
Member

mzur commented Apr 7, 2022

We should use multiple concurrent uploads if possible. It could be done like this: Track both the cumulated size and number of the files that are uploading at a given moment. If the size and number is below a given threshold (e.g. 50 MB and 5 files), immediately start uploading the next file. If not, wait for an upload to finish and check again. This way, concurrent uploads can be used to upload many smaller files but large files are uploaded one after the other.

@mzur mzur added the student label Apr 7, 2022
@mzur mzur moved this to Medium Priority in BIIGLE Roadmap Apr 7, 2022
@mzur
Copy link
Member Author

mzur commented May 11, 2022

This must take chunked files into account (see #9). Concurrent uploads should only be used when no chunked file is uploaded. This can probalby be solved by setting the concurrent upload size threshold (e.g. 50 MB) below the chunk size (e.g. 100 MB).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Medium Priority
Development

No branches or pull requests

1 participant