WX-927 GCP Batch: LogsPolicy.PATH now streams the logs to GCS

[AlexITC]

Description

Instead of pushing the logs file after the job completes, the logs are now streamed to GCS.

This is how it works:

• Mount the main GCS bucket as a disk in the VM filesystem.
• Configure Batch to store the logs in the mounted disk.
• The log file belongs to the same path used by the task files.

Notes:

• I haven't found a way to tail the GCS file but running cat continuously display the new logs.
• I'm not sure whether the logs are streamed live or when the runnable completes, if needed, I can evaluate this.
• There are some tricks I used to get this done, I'm open to suggestions for improving the approach.
• Follows up from WX-927 GCP Batch: LogsPolicy is now configurable #7491

Release Notes Confirmation

CHANGELOG.md

• I updated CHANGELOG.md in this PR
• I assert that this change shouldn't be included in CHANGELOG.md because it doesn't impact community users

Terra Release Notes

• I added a suggested release notes entry in this Jira ticket
• I assert that this change doesn't need Jira release notes because it doesn't impact Terra users

[mcovarr]

Hi Alex, in our sync meetings we discussed an aysnc "sidecar" type solution for log file syncing, much like PAPI v2 has now. We were concerned that mounting a GCS filesystem that includes potentially large task files might consume a lot of bandwidth as these files are written. Streaming of files other than task logs are not necessary for our use cases.

[dspeck1]

The mounted GCS bucket is only for logs. The outputs still go through delocalization. There is not an option to do a sidecar within GCP Batch. Google would have to enable it within the product. We have asked and the response was this to do it this way.

[mcovarr]

OK thank you for the clarification, we'll try it this way then. 😄

[mcovarr]

Unfortunately this approach seems to have some issues:

• The task log is not actually streamed as the WDL task runs. The "streaming" appears to happen at the level of GCP Batch runnables. While there is a lot of output in the task log from the runnables preceding the WDL task, there is no output uploaded from the WDL task itself until the task completes.
• The contents of the task log contain log formatting that is not part of the "raw" stdout or stderr.
• The contents of the task log also appear to contain GCP Batch agent log output intermingled with the task log output.
• There are extra files being uploaded with names like stdout-job-ab8e9cd9-2dc3-6e005a9d-a38f-441d00-group0-0.log and stderr-job-ab8e9cd9-2dc3-6e005a9d-a38f-441d00-group0-0.log that appear to be Batch agent logs.

In the course of evaluating this I discovered that the stdout and stderr files actually are being streamed correctly, and furthermore that this is implemented via "sidecar" runnables created from Cromwell. Since the task log should conceptually be the interleaved stdout and stderr, it seems this approach should also work for the task logs.

[mcovarr]

Sketch of stdout/stderr-like sidecar-ish "streaming" solution here, definitely not fully baked yet.

[AlexITC]

Thanks for sharing, do let me know if you need anything from me.

[mcovarr]

Thanks Alex. I just realized today that the approach in my PR does not capture output from any of the setup / localization / delocalization runnables. I'm not sure yet how much this matters, but it may turn out that the logging story isn't completely resolved yet. 🙂