Registry sync #3
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The skopeo sync command can sync images between a SOURCE and a destination. The purpose of this command is to assist with the mirroring of container images from different docker registries to a single docker registry. Right now the following transport matrix is implemented: * `docker://` -> `docker://` * `docker://` -> `dir:` * `dir:` -> `docker://` The `dir:` transport is supported to handle the use case of air-gapped environments. In this context users can perform an initial sync on a trusted machine connected to the internet; that would be a `docker://` -> `dir:` sync. The target directory can be copied to a removable drive that can then be plugged into a node of the air-gapped environment. From there a `dir:` -> `docker://` sync will import all the images into the registry serving the air-gapped environment. The image namespace is changed during the `docker://` to `docker://` or `dir:` copy. The FQDN of the registry hosting the image will be added as new root namespace of the image. For example, the image `registry.example.com/busybox:latest` will be copied to `registry.local.lan/registry.example.com/busybox:latest`. The image namespace is not changed when doing a `dir:` -> `docker://` sync operation. The alteration of the image namespace is used to nicely scope images coming from different registries (the Docker Hub, quay.io, gcr, other registries). That allows all of them to be hosted on the same registry without incurring in clashes and making their origin explicit. Signed-off-by: Flavio Castelli <[email protected]> Co-authored-by: Marco Vedovati <[email protected]>
Remove the $HOME/.docker directory when tearing down a cluster, so that subsequent cluster creations can be carried out successfully. Signed-off-by: Marco Vedovati <[email protected]>
Add sync integration tests, covering all the possible combinations of SOURCE,DEST. Signed-off-by: Marco Vedovati <[email protected]>
…ries catalog, so this is my attempt at making a contribution to this project for that purpose
The skopeo sync command can sync images between a SOURCE and a destination. The purpose of this command is to assist with the mirroring of container images from different docker registries to a single docker registry. Right now the following transport matrix is implemented: * `docker://` -> `docker://` * `docker://` -> `dir:` * `dir:` -> `docker://` The `dir:` transport is supported to handle the use case of air-gapped environments. In this context users can perform an initial sync on a trusted machine connected to the internet; that would be a `docker://` -> `dir:` sync. The target directory can be copied to a removable drive that can then be plugged into a node of the air-gapped environment. From there a `dir:` -> `docker://` sync will import all the images into the registry serving the air-gapped environment. The image namespace is changed during the `docker://` to `docker://` or `dir:` copy. The FQDN of the registry hosting the image will be added as new root namespace of the image. For example, the image `registry.example.com/busybox:latest` will be copied to `registry.local.lan/registry.example.com/busybox:latest`. The image namespace is not changed when doing a `dir:` -> `docker://` sync operation. The alteration of the image namespace is used to nicely scope images coming from different registries (the Docker Hub, quay.io, gcr, other registries). That allows all of them to be hosted on the same registry without incurring in clashes and making their origin explicit. Signed-off-by: Flavio Castelli <[email protected]> Co-authored-by: Marco Vedovati <[email protected]>
Remove the $HOME/.docker directory when tearing down a cluster, so that subsequent cluster creations can be carried out successfully. Signed-off-by: Marco Vedovati <[email protected]>
Add sync integration tests, covering all the possible combinations of SOURCE,DEST. Signed-off-by: Marco Vedovati <[email protected]>
…ries catalog, so this is my attempt at making a contribution to this project for that purpose
… increment happens before the continue
…nc, next step is to inspect and compare
flavio
force-pushed
the
sync
branch
6 times, most recently
from
7d22124
to
3bec128
Compare
Have to remove manifest just before the copy
|
A friendly reminder that this PR had no activity for 30 days. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I needed the sync to pull once, so this will read the target directory and check for an existing manifest.json and ensure that the upstream has a different version prior to the copy. Also introduced go routines to thread out the compare and copy, my case has me whole sale cloning a registry, to an air gapped enclave (over 900 Repositories and counting).
Some things that are missing from this solution is Multi-arch copies. Found out the hard way that any image push with the same tag, even different arch, will over write existing in the target registry. Still looking for solutions to this problem