Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: conditions API #129

Open
alsenz opened this issue Jul 26, 2024 · 0 comments
Open

Feature request: conditions API #129

alsenz opened this issue Jul 26, 2024 · 0 comments

Comments

@alsenz
Copy link

alsenz commented Jul 26, 2024

Firstly thank you for this library I've just started using it and the experience is really good.

Many object stores now support conditional gets and writes. The former allows better caching, and the latter allows developers to write data to an object if and only if the object hasn't been changed since a previous point in time, which forms the backbone for 'atomic' updates to datasets on object storage (a reference use case would be databricks delta format).

Generally object store providers either provide this functionality using a conditional header or payload containing an etag, a version or generation or even a last modified.

Examples:
GCS: https://pkg.go.dev/cloud.google.com/go/storage#hdr-Conditions
Azure: https://learn.microsoft.com/en-us/rest/api/storageservices/specifying-conditional-headers-for-blob-service-operations

Note - (infamously?) AWS S3 doesn't support this. The standard response to this seems to be to ask library users to implement their own locking provider, and not address this in the library.

Here is an example of another library that has nice API for this: https://docs.rs/object_store/latest/object_store/struct.UpdateVersion.html

My thought would be that the ObjectAttributes API could include this metadata, and then I'm not sure how the API would be be changed conservatively to support this for Get or Upload; I quite like the minimal addition of an If(condition) receiver function that returns a wrapped object (similar to the cloud-storage library).

If this feature gets the maintainers' blessing I am happy to contribute pull requests for GCS and a suitable-for-testing purposes version for LocalFilesystem.

Thanks again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant