Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flake DecompressiveTranscodingIntegrationTest.WriteAndReadCompressedJson retry exhausted #14543

Open
scotthart opened this issue Jul 23, 2024 · 0 comments
Open

Flake DecompressiveTranscodingIntegrationTest.WriteAndReadCompressedJson retry exhausted #14543

scotthart opened this issue Jul 23, 2024 · 0 comments
Labels
api: storage Issues related to the Cloud Storage API. type: cleanup An internal cleanup or hygiene concern.

Comments

@scotthart
Copy link
Member

https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/88d5094feed3c301579055c71ba16e9d6f21cff9/fedora-latest-bazel-coverage-__default__/log-ae508921-4547-4265-9abd-b1d153132734.txt

[ RUN      ] DecompressiveTranscodingIntegrationTest.WriteAndReadCompressedJson
google/cloud/storage/tests/decompressive_transcoding_integration_test.cc:107: Failure
Value of: insert
Expected: code is equal to OK and message is anything
  Actual: 928-byte object <60-51 C7-01 00-00 00-00 00-AD 29-36 10-7F 00-00 80-9F AD-58 FC-7F 00-00 10-20 00-00 00-00 00-00 80-9F AD-58 FC-7F 00-00 C0-BA 3D-36 10-7F 00-00 10-20 00-00 00-00 00-00 B0-F8 FF-FF FF-FF FF-FF ... 08-00 00-00 10-7F 00-00 20-A2 AD-58 FC-7F 00-00 00-3F 67-36 10-7F 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 F0-A0 AD-58 FC-7F 00-00 20-39 67-36 10-7F 00-00 B0-38 67-36 10-7F 00-00>, whose status is DEADLINE_EXCEEDED: Retry policy exhausted, with a last message of timeout [3s] while waiting for Close() error_info={reason=DEADLINE_EXCEEDED, domain=gcloud-cpp, metadata={gcloud-cpp.retry.on-entry=false, gcloud-cpp.retry.function=InsertObjectMedia, gcloud-cpp.source.function=TimeoutError, gcloud-cpp.source.line=121, gcloud-cpp.source.filename=google/cloud/storage/internal/grpc/stub.cc, gcloud-cpp.retry.reason=retry-policy-exhausted, gcloud-cpp.retry.original-message=timeout [3s] while waiting for Close(), gcloud-cpp.version=v2.27.0-rc}}, with a code that isn't equal to OK, but a message that is anything

2024-07-23T14:49:21.524026024Z [INFO] <139707604016640> Enabling logging for raw-client (google/cloud/storage/client_options.cc:238)
2024-07-23T14:49:21.524066719Z [INFO] <139707604016640> Enabling logging for rpc (google/cloud/storage/client_options.cc:238)
2024-07-23T14:49:21.524075012Z [INFO] <139707604016640> Enabling logging for rpc-streams (google/cloud/storage/client_options.cc:238)
2024-07-23T14:49:21.524761456Z [INFO] <139707604016640> Enabling logging for raw-client (google/cloud/storage/client_options.cc:238)
2024-07-23T14:49:21.524779687Z [INFO] <139707604016640> Enabling logging for rpc (google/cloud/storage/client_options.cc:238)
2024-07-23T14:49:21.524786217Z [INFO] <139707604016640> Enabling logging for rpc-streams (google/cloud/storage/client_options.cc:238)
2024-07-23T14:49:21.591627635Z [INFO] <139707604016640> Enabled logging for gRPC calls (google/cloud/storage/internal/storage_stub_factory.cc:91)
2024-07-23T14:49:21.591755124Z [INFO] <139707604016640> Refreshing channel [0] (google/cloud/storage/internal/grpc/channel_refresh.cc:50)
2024-07-23T14:49:21.598533452Z [INFO] <139707578459840> Refreshing channel [0] (google/cloud/storage/internal/grpc/channel_refresh.cc:50)
2024-07-23T14:49:21.600632718Z [INFO] <139707578459840> Refreshing channel [0] (google/cloud/storage/internal/grpc/channel_refresh.cc:50)
2024-07-23T14:49:21.669618256Z [DEBUG] <139707604016640> WriteObject(1) (google/cloud/storage/internal/storage_logging_decorator.cc:296)
2024-07-23T14:49:21.773004529Z [DEBUG] <139707604016640> Write(1)() << google.storage.v2.WriteObjectRequest { write_object_spec { resource { name: "ob-4EghsE3nTfStI8PxB4K6XJ9KNofucZpOkpGywderXStBZHUibtce9tzSaoilHuoJJVcgHJeGDBr9EaQpB2Eu5XeDEewbDekG6QwjgKvs7MX3R8GTbVTXisXXYpDzn...<truncated>..." bucket: "projects/_/buckets/gcs-grpc-team-cloud-cpp-testing-bucket" content_encoding: "gzip" content_type: "text/plain" } if_generation_match: 0 } checksummed_data { content: "\037\213\010\000\000\000\000\000\000\003\234\335\313\0254={\226\3219Q|!\274\317Q\022q\220\000`s\206\037\014\006C\364T/\210`\367\240\'\275\356\3315\332]%\375\371\363\377>\361\317\377\372\027\377\356\357\377\372o\377\370\357\377\365\177\374\353_\375\303\337\376\327\177\371\353\337\374\355\237\376\372\017\377\370\237\377\353\177\377\353o\377\363\357\377\341\257\377\361\375\374\237\376\345\377\371\337\177\375\335\337\376\355?\373\377\253?...<truncated>..." crc32c: 2202592185 } object_checksums { crc32c: 2202592185 } finish_write: true } (./google/cloud/internal/streaming_write_rpc_logging.h:56)
2024-07-23T14:49:21.775071576Z [DEBUG] <139707604016640> Write(1)() >> true (./google/cloud/internal/streaming_write_rpc_logging.h:59)
2024-07-23T14:49:21.775958925Z [DEBUG] <139707604016640> Close(1)() << (void) (./google/cloud/internal/streaming_write_rpc_logging.h:65)
2024-07-23T14:49:24.778622740Z [DEBUG] <139707578459840> Cancel(1)() << (void) (./google/cloud/internal/streaming_write_rpc_logging.h:49)
2024-07-23T14:49:24.780225950Z [DEBUG] <139707604016640> Close(1)() >> CANCELLED: CANCELLED (./google/cloud/internal/streaming_write_rpc_logging.h:71)
2024-07-23T14:49:24.780520658Z [DEBUG] <139707578459840> Cancel(1)() >> (void) (./google/cloud/internal/streaming_write_rpc_logging.h:51)
2024-07-23T14:49:24.800089526Z [DEBUG] <139707604016640> WriteObject(2) (google/cloud/storage/internal/storage_logging_decorator.cc:296)
2024-07-23T14:49:24.802416051Z [DEBUG] <139707604016640> Write(2)() << google.storage.v2.WriteObjectRequest { write_object_spec { resource { name: "ob-4EghsE3nTfStI8PxB4K6XJ9KNofucZpOkpGywderXStBZHUibtce9tzSaoilHuoJJVcgHJeGDBr9EaQpB2Eu5XeDEewbDekG6QwjgKvs7MX3R8GTbVTXisXXYpDzn...<truncated>..." bucket: "projects/_/buckets/gcs-grpc-team-cloud-cpp-testing-bucket" content_encoding: "gzip" content_type: "text/plain" } if_generation_match: 0 } checksummed_data { content: "\037\213\010\000\000\000\000\000\000\003\234\335\313\0254={\226\3219Q|!\274\317Q\022q\220\000`s\206\037\014\006C\364T/\210`\367\240\'\275\356\3315\332]%\375\371\363\377>\361\317\377\372\027\377\356\357\377\372o\377\370\357\377\365\177\374\353_\375\303\337\376\327\177\371\353\337\374\355\237\376\372\017\377\370\237\377\353\177\377\353o\377\363\357\377\341\257\377\361\375\374\237\376\345\377\371\337\177\375\335\337\376\355?\373\377\253?...<truncated>..." crc32c: 2202592185 } object_checksums { crc32c: 2202592185 } finish_write: true } (./google/cloud/internal/streaming_write_rpc_logging.h:56)
2024-07-23T14:49:24.803112996Z [DEBUG] <139707604016640> Write(2)() >> true (./google/cloud/internal/streaming_write_rpc_logging.h:59)
2024-07-23T14:49:24.803989069Z [DEBUG] <139707604016640> Close(2)() << (void) (./google/cloud/internal/streaming_write_rpc_logging.h:65)
2024-07-23T14:49:27.806144628Z [DEBUG] <139707578459840> Cancel(2)() << (void) (./google/cloud/internal/streaming_write_rpc_logging.h:49)
2024-07-23T14:49:27.807482751Z [DEBUG] <139707578459840> Cancel(2)() >> (void) (./google/cloud/internal/streaming_write_rpc_logging.h:51)
2024-07-23T14:49:27.807785533Z [DEBUG] <139707604016640> Close(2)() >> CANCELLED: CANCELLED (./google/cloud/internal/streaming_write_rpc_logging.h:71)
2024-07-23T14:49:27.829024669Z [DEBUG] <139707604016640> WriteObject(3) (google/cloud/storage/internal/storage_logging_decorator.cc:296)
2024-07-23T14:49:27.831496904Z [DEBUG] <139707604016640> Write(3)() << google.storage.v2.WriteObjectRequest { write_object_spec { resource { name: "ob-4EghsE3nTfStI8PxB4K6XJ9KNofucZpOkpGywderXStBZHUibtce9tzSaoilHuoJJVcgHJeGDBr9EaQpB2Eu5XeDEewbDekG6QwjgKvs7MX3R8GTbVTXisXXYpDzn...<truncated>..." bucket: "projects/_/buckets/gcs-grpc-team-cloud-cpp-testing-bucket" content_encoding: "gzip" content_type: "text/plain" } if_generation_match: 0 } checksummed_data { content: "\037\213\010\000\000\000\000\000\000\003\234\335\313\0254={\226\3219Q|!\274\317Q\022q\220\000`s\206\037\014\006C\364T/\210`\367\240\'\275\356\3315\332]%\375\371\363\377>\361\317\377\372\027\377\356\357\377\372o\377\370\357\377\365\177\374\353_\375\303\337\376\327\177\371\353\337\374\355\237\376\372\017\377\370\237\377\353\177\377\353o\377\363\357\377\341\257\377\361\375\374\237\376\345\377\371\337\177\375\335\337\376\355?\373\377\253?...<truncated>..." crc32c: 2202592185 } object_checksums { crc32c: 2202592185 } finish_write: true } (./google/cloud/internal/streaming_write_rpc_logging.h:56)
2024-07-23T14:49:27.832215499Z [DEBUG] <139707604016640> Write(3)() >> true (./google/cloud/internal/streaming_write_rpc_logging.h:59)
2024-07-23T14:49:27.833025303Z [DEBUG] <139707604016640> Close(3)() << (void) (./google/cloud/internal/streaming_write_rpc_logging.h:65)
2024-07-23T14:49:30.835764565Z [DEBUG] <139707578459840> Cancel(3)() << (void) (./google/cloud/internal/streaming_write_rpc_logging.h:49)
2024-07-23T14:49:30.836855712Z [DEBUG] <139707604016640> Close(3)() >> CANCELLED: CANCELLED (./google/cloud/internal/streaming_write_rpc_logging.h:71)
2024-07-23T14:49:30.837159539Z [DEBUG] <139707578459840> Cancel(3)() >> (void) (./google/cloud/internal/streaming_write_rpc_logging.h:51)
2024-07-23T14:49:30.892531008Z [DEBUG] <139707604016640> WriteObject(4) (google/cloud/storage/internal/storage_logging_decorator.cc:296)
2024-07-23T14:49:30.895037515Z [DEBUG] <139707604016640> Write(4)() << google.storage.v2.WriteObjectRequest { write_object_spec { resource { name: "ob-4EghsE3nTfStI8PxB4K6XJ9KNofucZpOkpGywderXStBZHUibtce9tzSaoilHuoJJVcgHJeGDBr9EaQpB2Eu5XeDEewbDekG6QwjgKvs7MX3R8GTbVTXisXXYpDzn...<truncated>..." bucket: "projects/_/buckets/gcs-grpc-team-cloud-cpp-testing-bucket" content_encoding: "gzip" content_type: "text/plain" } if_generation_match: 0 } checksummed_data { content: "\037\213\010\000\000\000\000\000\000\003\234\335\313\0254={\226\3219Q|!\274\317Q\022q\220\000`s\206\037\014\006C\364T/\210`\367\240\'\275\356\3315\332]%\375\371\363\377>\361\317\377\372\027\377\356\357\377\372o\377\370\357\377\365\177\374\353_\375\303\337\376\327\177\371\353\337\374\355\237\376\372\017\377\370\237\377\353\177\377\353o\377\363\357\377\341\257\377\361\375\374\237\376\345\377\371\337\177\375\335\337\376\355?\373\377\253?...<truncated>..." crc32c: 2202592185 } object_checksums { crc32c: 2202592185 } finish_write: true } (./google/cloud/internal/streaming_write_rpc_logging.h:56)
2024-07-23T14:49:30.895758594Z [DEBUG] <139707604016640> Write(4)() >> true (./google/cloud/internal/streaming_write_rpc_logging.h:59)
2024-07-23T14:49:30.896472903Z [DEBUG] <139707604016640> Close(4)() << (void) (./google/cloud/internal/streaming_write_rpc_logging.h:65)
2024-07-23T14:49:33.899648609Z [DEBUG] <139707578459840> Cancel(4)() << (void) (./google/cloud/internal/streaming_write_rpc_logging.h:49)
2024-07-23T14:49:33.900622914Z [DEBUG] <139707604016640> Close(4)() >> CANCELLED: CANCELLED (./google/cloud/internal/streaming_write_rpc_logging.h:71)
2024-07-23T14:49:33.900848060Z [DEBUG] <139707578459840> Cancel(4)() >> (void) (./google/cloud/internal/streaming_write_rpc_logging.h:51)
2024-07-23T14:49:33.997188709Z [DEBUG] <139707604016640> WriteObject(5) (google/cloud/storage/internal/storage_logging_decorator.cc:296)
2024-07-23T14:49:33.999538721Z [DEBUG] <139707604016640> Write(5)() << google.storage.v2.WriteObjectRequest { write_object_spec { resource { name: "ob-4EghsE3nTfStI8PxB4K6XJ9KNofucZpOkpGywderXStBZHUibtce9tzSaoilHuoJJVcgHJeGDBr9EaQpB2Eu5XeDEewbDekG6QwjgKvs7MX3R8GTbVTXisXXYpDzn...<truncated>..." bucket: "projects/_/buckets/gcs-grpc-team-cloud-cpp-testing-bucket" content_encoding: "gzip" content_type: "text/plain" } if_generation_match: 0 } checksummed_data { content: "\037\213\010\000\000\000\000\000\000\003\234\335\313\0254={\226\3219Q|!\274\317Q\022q\220\000`s\206\037\014\006C\364T/\210`\367\240\'\275\356\3315\332]%\375\371\363\377>\361\317\377\372\027\377\356\357\377\372o\377\370\357\377\365\177\374\353_\375\303\337\376\327\177\371\353\337\374\355\237\376\372\017\377\370\237\377\353\177\377\353o\377\363\357\377\341\257\377\361\375\374\237\376\345\377\371\337\177\375\335\337\376\355?\373\377\253?...<truncated>..." crc32c: 2202592185 } object_checksums { crc32c: 2202592185 } finish_write: true } (./google/cloud/internal/streaming_write_rpc_logging.h:56)
2024-07-23T14:49:34.000209878Z [DEBUG] <139707604016640> Write(5)() >> true (./google/cloud/internal/streaming_write_rpc_logging.h:59)
2024-07-23T14:49:34.001021345Z [DEBUG] <139707604016640> Close(5)() << (void) (./google/cloud/internal/streaming_write_rpc_logging.h:65)
2024-07-23T14:49:37.003672956Z [DEBUG] <139707578459840> Cancel(5)() << (void) (./google/cloud/internal/streaming_write_rpc_logging.h:49)
2024-07-23T14:49:37.004767711Z [DEBUG] <139707604016640> Close(5)() >> CANCELLED: CANCELLED (./google/cloud/internal/streaming_write_rpc_logging.h:71)
2024-07-23T14:49:37.005015305Z [DEBUG] <139707578459840> Cancel(5)() >> (void) (./google/cloud/internal/streaming_write_rpc_logging.h:51)
2024-07-23T14:49:37.228534564Z [DEBUG] <139707604016640> WriteObject(6) (google/cloud/storage/internal/storage_logging_decorator.cc:296)
2024-07-23T14:49:37.230826758Z [DEBUG] <139707604016640> Write(6)() << google.storage.v2.WriteObjectRequest { write_object_spec { resource { name: "ob-4EghsE3nTfStI8PxB4K6XJ9KNofucZpOkpGywderXStBZHUibtce9tzSaoilHuoJJVcgHJeGDBr9EaQpB2Eu5XeDEewbDekG6QwjgKvs7MX3R8GTbVTXisXXYpDzn...<truncated>..." bucket: "projects/_/buckets/gcs-grpc-team-cloud-cpp-testing-bucket" content_encoding: "gzip" content_type: "text/plain" } if_generation_match: 0 } checksummed_data { content: "\037\213\010\000\000\000\000\000\000\003\234\335\313\0254={\226\3219Q|!\274\317Q\022q\220\000`s\206\037\014\006C\364T/\210`\367\240\'\275\356\3315\332]%\375\371\363\377>\361\317\377\372\027\377\356\357\377\372o\377\370\357\377\365\177\374\353_\375\303\337\376\327\177\371\353\337\374\355\237\376\372\017\377\370\237\377\353\177\377\353o\377\363\357\377\341\257\377\361\375\374\237\376\345\377\371\337\177\375\335\337\376\355?\373\377\253?...<truncated>..." crc32c: 2202592185 } object_checksums { crc32c: 2202592185 } finish_write: true } (./google/cloud/internal/streaming_write_rpc_logging.h:56)
2024-07-23T14:49:37.231474087Z [DEBUG] <139707604016640> Write(6)() >> true (./google/cloud/internal/streaming_write_rpc_logging.h:59)
2024-07-23T14:49:37.232184056Z [DEBUG] <139707604016640> Close(6)() << (void) (./google/cloud/internal/streaming_write_rpc_logging.h:65)
2024-07-23T14:49:40.234490486Z [DEBUG] <139707578459840> Cancel(6)() << (void) (./google/cloud/internal/streaming_write_rpc_logging.h:49)
2024-07-23T14:49:40.235436087Z [DEBUG] <139707604016640> Close(6)() >> CANCELLED: CANCELLED (./google/cloud/internal/streaming_write_rpc_logging.h:71)
2024-07-23T14:49:40.235708497Z [DEBUG] <139707578459840> Cancel(6)() >> (void) (./google/cloud/internal/streaming_write_rpc_logging.h:51)
2024-07-23T14:49:40.267903231Z [INFO] <139707604016640> Enabling logging for raw-client (google/cloud/storage/client_options.cc:238)
2024-07-23T14:49:40.267930565Z [INFO] <139707604016640> Enabling logging for rpc (google/cloud/storage/client_options.cc:238)
2024-07-23T14:49:40.267936035Z [INFO] <139707604016640> Enabling logging for rpc-streams (google/cloud/storage/client_options.cc:238)
2024-07-23T14:49:40.268309712Z [INFO] <139707604016640> Enabling logging for raw-client (google/cloud/storage/client_options.cc:238)
2024-07-23T14:49:40.268321309Z [INFO] <139707604016640> Enabling logging for rpc (google/cloud/storage/client_options.cc:238)
2024-07-23T14:49:40.268325875Z [INFO] <139707604016640> Enabling logging for rpc-streams (google/cloud/storage/client_options.cc:238)
2024-07-23T14:49:40.287754359Z [INFO] <139707604016640> Enabled logging for gRPC calls (google/cloud/storage/internal/storage_stub_factory.cc:91)
2024-07-23T14:49:40.287812090Z [INFO] <139707604016640> Refreshing channel [0] (google/cloud/storage/internal/grpc/channel_refresh.cc:50)
2024-07-23T14:49:40.291017404Z [INFO] <139706108974784> Refreshing channel [0] (google/cloud/storage/internal/grpc/channel_refresh.cc:50)
2024-07-23T14:49:40.292889202Z [INFO] <139706108974784> Refreshing channel [0] (google/cloud/storage/internal/grpc/channel_refresh.cc:50)
[  FAILED  ] DecompressiveTranscodingIntegrationTest.WriteAndReadCompressedJson (18836 ms)
@scotthart scotthart added api: storage Issues related to the Cloud Storage API. type: cleanup An internal cleanup or hygiene concern. labels Jul 23, 2024
@coryan coryan self-assigned this Jul 23, 2024
@coryan coryan mentioned this issue Jul 23, 2024
Merged
@coryan coryan removed their assignment Jul 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
api: storage Issues related to the Cloud Storage API. type: cleanup An internal cleanup or hygiene concern.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@coryan @scotthart