Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking bug for metadata-only-import from S3 or other hosted storage #905

Open
olsen232 opened this issue Sep 7, 2023 · 1 comment
Open
Labels
enhancement New feature or request meta Meta issue for a larger set of issues/PRs/tasks

Comments

@olsen232
Copy link
Collaborator

olsen232 commented Sep 7, 2023

A yet-to-be-fully-designed flow that creates a tile-based Kart dataset that is "backed" in some sense by the original data on S3 (or similar), rather than being necessarily stored in the local Kart repo or available-on-demand from the remote Kart repo. The storage format will be almost identical to a regular tile-based Kart dataset, except that extra information will be stored about the original location where the tile was found, so it can be fetched from there if necessary.

@olsen232
Copy link
Collaborator Author

olsen232 commented Oct 10, 2023

Still TODO:

  • Add a per-dataset "no checkout" config option
    • Expose which datasets are non-checkout in kart status
  • Don't fetch entire PC tiles just to extract metadata
  • Fix kart lfs+ gc to clean files that were fetched from S3
  • Make sure BYOD files are never pushed to LFS remote
  • Rename byod-import commands
  • add a remote-storage.json meta-item that allows for S3 fetches to be redirected
  • add config options that allows for S3 fetches to be redirected

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request meta Meta issue for a larger set of issues/PRs/tasks
Projects
None yet
Development

No branches or pull requests

1 participant