Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

需求：oss 基于多线程逻辑分片 #12

Open
quhongwei opened this issue Oct 5, 2018 · 3 comments
Open

需求：oss 基于多线程逻辑分片 #12

quhongwei opened this issue Oct 5, 2018 · 3 comments
Labels
enhancement New feature or request

Comments

@quhongwei
Copy link
Member

oss 基于多线程逻辑分片，提升大文件逻辑分片的效率
@quhongwei quhongwei added the enhancement New feature or request label Oct 5, 2018
@michaelhuang
Copy link

感觉这个没必要做，多大文件算提高呢，比如1T文件，单机再怎么多线程其实提高并不多，还是得集群处理；比如先按照10G拆分，生成分片任务，然后再次递归，对每个10G按照1G，生成分片任务，以此类推，到每片10M终止

@quhongwei
Copy link
Member Author

感觉这个没必要做，多大文件算提高呢，比如1T文件，单机再怎么多线程其实提高并不多，还是得集群处理；比如先按照10G拆分，生成分片任务，然后再次递归，对每个10G按照1G，生成分片任务，以此类推，到每片10M终止

逻辑分片的目的就是为了集群处理分片文件， 大文件一分为十只能一个线程做， 十分分一百可以多个线程并发处理，

@michaelhuang
Copy link

明白了，类似reactor模型，boss单线程获得一个请求，先预处理下【大文件一分为十】; 然后交给work Threadpool【十分分一百】👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@michaelhuang @quhongwei