You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Firstly thank you for this library I've just started using it and the experience is really good.
Many object stores now support conditional gets and writes. The former allows better caching, and the latter allows developers to write data to an object if and only if the object hasn't been changed since a previous point in time, which forms the backbone for 'atomic' updates to datasets on object storage (a reference use case would be databricks delta format).
Generally object store providers either provide this functionality using a conditional header or payload containing an etag, a version or generation or even a last modified.
Note - (infamously?) AWS S3 doesn't support this. The standard response to this seems to be to ask library users to implement their own locking provider, and not address this in the library.
My thought would be that the ObjectAttributes API could include this metadata, and then I'm not sure how the API would be be changed conservatively to support this for Get or Upload; I quite like the minimal addition of an If(condition) receiver function that returns a wrapped object (similar to the cloud-storage library).
If this feature gets the maintainers' blessing I am happy to contribute pull requests for GCS and a suitable-for-testing purposes version for LocalFilesystem.
Thanks again.
The text was updated successfully, but these errors were encountered:
Firstly thank you for this library I've just started using it and the experience is really good.
Many object stores now support conditional gets and writes. The former allows better caching, and the latter allows developers to write data to an object if and only if the object hasn't been changed since a previous point in time, which forms the backbone for 'atomic' updates to datasets on object storage (a reference use case would be databricks delta format).
Generally object store providers either provide this functionality using a conditional header or payload containing an etag, a version or generation or even a last modified.
Examples:
GCS: https://pkg.go.dev/cloud.google.com/go/storage#hdr-Conditions
Azure: https://learn.microsoft.com/en-us/rest/api/storageservices/specifying-conditional-headers-for-blob-service-operations
Note - (infamously?) AWS S3 doesn't support this. The standard response to this seems to be to ask library users to implement their own locking provider, and not address this in the library.
Here is an example of another library that has nice API for this: https://docs.rs/object_store/latest/object_store/struct.UpdateVersion.html
My thought would be that the ObjectAttributes API could include this metadata, and then I'm not sure how the API would be be changed conservatively to support this for Get or Upload; I quite like the minimal addition of an If(condition) receiver function that returns a wrapped object (similar to the cloud-storage library).
If this feature gets the maintainers' blessing I am happy to contribute pull requests for GCS and a suitable-for-testing purposes version for LocalFilesystem.
Thanks again.
The text was updated successfully, but these errors were encountered: