Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PATCH: read {path}: file already closed error following update of go-containerregistry to v0.15.2 #2019

Open
parvathikp100 opened this issue Oct 10, 2024 · 3 comments
Labels
bug Something isn't working

Comments

@parvathikp100
Copy link

parvathikp100 commented Oct 10, 2024

Describe the bug

We recently upgraded our go-containerregistry version from v0.11.0 to v0.15.2 and there has been a sudden surge in the number of the following

Patch "https://xxx/blobs/uploads/xxx?_state=xxx%3D%3D": read /xxx/oci/blobs/sha256/xxx: file already closed

error which seem to be originating from the http.PATCH request in streamBlob function

Also interesting is to note that similar surge in errors have been seen in http requests being made from commitManifest and initiateUpload

The major change that is common to streamBlob, initiateUpload and commitManifest is the addition of retry.Never(ctx) which seems to be affecting the retries somehow?

This is the PR that introduced it - #1502 (comment)

To Reproduce

No clue on reproducing it yet, problem seems to be insufficient retries if request fails due to some transient issue because these requests have passed sometimes for the same blobs

Expected behavior

Blobs are written successfully without errors at any of the http requests

Additional context

Added a comment on the original PR with some more details - #1502 (comment)

  • Output of crane version
  • Registry used (e.g., GCR, ECR, Quay)
@parvathikp100 parvathikp100 added the bug Something isn't working label Oct 10, 2024
@parvathikp100
Copy link
Author

I also added a comment on this PR that introduced the retry predicate - #1502 (comment).

@parvathikp100
Copy link
Author

Also see a rise in errors for other http requests originating from uploadOne():

Put "https://xxx/manifests/sha256:xxx": http: ContentLength=2706 with Body length 0
PUT https://xxx/blobs/uploads/xxx?_state=REDACTED&digest=sha256%3Axxx: DIGEST_INVALID: provided digest did not match uploaded content; map[Digest:sha256:xxx Reason:map[]]

@parvathikp100
Copy link
Author

@jonjohnsonjr can you help look into this? Do you think our suspicions regarding what could have caused this surge in failures is valid?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant