Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make I/O asynchronous per partition #732

Open
kordejong opened this issue Nov 14, 2024 · 0 comments
Open

Make I/O asynchronous per partition #732

kordejong opened this issue Nov 14, 2024 · 0 comments

Comments

@kordejong
Copy link
Member

Current approach is:

  • Group partitions per locality
  • Iterate over localities:
    • Read or write the partitions
    • Wait for this to finish

This works fine for serial I/O. No need to hammer at the controller from multiple processes.

Drawbacks are:

  • A partition that is read can participate in a computation, overlapping I/O of other partitions
  • A ready partition can be written without waiting for other partitions, overlapping computations of other partitions
  • Parallel I/O (independent) from multiple processes is fine
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant