Job batching and limit for requests with many files

[mzur]

We should implement a limit to the maximum number of files per storage request. On biigle.de, we have a user who created several requests with multiple 10k or even more than 100k images. Maybe we could implement a 10k limit. If users want to upload more, they have to chunk the files into several storage requests.

One issue that too many files can cause is too long run times for the queue jobs. Also they can theoretically spam the service with millions of small files (as long as the total size is within their quota but that's easy).

[dlangenk]

Wouldn't it make more sense to chunk the storage request into multiple queue jobs instead of limiting it on the user site? In the end both solutions would result in the same outcome (if the user submits multiple requests in your case), but having similar data as one storage request seems more manageable.

[mzur]

That's also a good idea! However, I think we need some kind of limit in any case. Maybe a higher one, then (100k, 500k)?

The ApproveStorageRequest job can be split up into several smaller jobs. The copying can be done as batched jobs (each copies 10k files) then the user is notified and then the pending directory is deleted.