Add the ability to set tail_from_end when gathering logs from volumes (…

…#984) Signed-off-by: Pete Wall <[email protected]>
grafana · Dec 10, 2024 · fdb83c0 · fdb83c0
1 parent ba700f9
commit fdb83c0
Show file tree

Hide file tree

Showing 23 changed files with 80 additions and 39 deletions.
diff --git a/.github/workflows/platform-test.yml b/.github/workflows/platform-test.yml
@@ -8,7 +8,7 @@ on:
       - 'charts/**'
       - '!charts/k8s-monitoring-v1/**'
   pull_request:
-    types: [ labeled ]
+    types: [labeled]
     paths:
       - 'charts/**'
       - '!charts/k8s-monitoring-v1/**'

diff --git a/charts/feature-cluster-metrics/README.md b/charts/feature-cluster-metrics/README.md
@@ -17,30 +17,30 @@ not on the list will be dropped. With [metrics tuning](#metrics-tuning--allow-li
 
 This chart includes the ability to collect metrics from the following:
 
-* The Kubernetes cluster itself
-* Sources like the Kubelet and cAdvisor
-* Common supporting services like [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) and
-[Node Exporter](https://github.com/prometheus/node_exporter)
-* Systems to capture additional data like Kepler
+*   The Kubernetes cluster itself
+*   Sources like the Kubelet and cAdvisor
+*   Common supporting services like [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) and
+    [Node Exporter](https://github.com/prometheus/node_exporter)
+*   Systems to capture additional data like Kepler
 
 ### Metrics sources
 
 The Cluster Metrics feature of the Kubernetes Observability Helm chart includes the following metric systems and
 their default allow lists:
 
-| Metric source | Gathers information about | Allow list |
-| --- | --- | --- |
-| API Server | Kubernetes API Server | NA |
-| [cAdvisor](https://github.com/google/cadvisor)| Containers on each node | [default-allow-lists/cadvisor.yaml](./default-allow-lists/cadvisor.yaml) |
-| [Kepler](https://sustainable-computing.io/) | Kubernetes cluster | [default-allow-lists/kepler.yaml](./default-allow-lists/kepler.yaml) |
-| Kube Controller Manager | Kubernetes Controller Manager | NA |
-| Kube Proxy | Kube Proxy | NA|
-| Kube Scheduler | Kube Scheduler |  NA|
-| Kubelet | Kubernetes information on each node | [default-allow-lists/kubelet.yaml](./default-allow-lists/kubelet.yaml) |
-| [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) |Kubernetes
-resources inside the cluster | [default-allow-lists/kube-state-metrics.yaml](./default-allow-lists/kube-state-metrics.yaml) |
-| [Node Exporter](https://github.com/prometheus/node_exporter) | Linux Kubernetes nodes | [default-allow-lists/node-exporter.yaml](./default-allow-lists/node-exporter.yaml), [default-allow-lists/node-exporter-integration.yaml](./default-allow-lists/node-exporter-integration.yaml)  |
-| [Windows Exporter](https://github.com/prometheus-community/windows_exporter) |  Windows Kubernetes nodes | [default-allow-lists/windows-exporter.yaml](./default-allow-lists/windows-exporter.yaml) |
+| Metric source                                                                | Gathers information about                                                                    | Allow list                                                                                                                                                                                     |
+|------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| API Server                                                                   | Kubernetes API Server                                                                        | NA                                                                                                                                                                                             |
+| [cAdvisor](https://github.com/google/cadvisor)                               | Containers on each node                                                                      | [default-allow-lists/cadvisor.yaml](./default-allow-lists/cadvisor.yaml)                                                                                                                       |
+| [Kepler](https://sustainable-computing.io/)                                  | Kubernetes cluster                                                                           | [default-allow-lists/kepler.yaml](./default-allow-lists/kepler.yaml)                                                                                                                           |
+| Kube Controller Manager                                                      | Kubernetes Controller Manager                                                                | NA                                                                                                                                                                                             |
+| Kube Proxy                                                                   | Kube Proxy                                                                                   | NA                                                                                                                                                                                             |
+| Kube Scheduler                                                               | Kube Scheduler                                                                               | NA                                                                                                                                                                                             |
+| Kubelet                                                                      | Kubernetes information on each node                                                          | [default-allow-lists/kubelet.yaml](./default-allow-lists/kubelet.yaml)                                                                                                                         |
+| [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics)       | Kubernetes                                                                                   |                                                                                                                                                                                                |
+| resources inside the cluster                                                 | [default-allow-lists/kube-state-metrics.yaml](./default-allow-lists/kube-state-metrics.yaml) |                                                                                                                                                                                                |
+| [Node Exporter](https://github.com/prometheus/node_exporter)                 | Linux Kubernetes nodes                                                                       | [default-allow-lists/node-exporter.yaml](./default-allow-lists/node-exporter.yaml), [default-allow-lists/node-exporter-integration.yaml](./default-allow-lists/node-exporter-integration.yaml) |
+| [Windows Exporter](https://github.com/prometheus-community/windows_exporter) | Windows Kubernetes nodes                                                                     | [default-allow-lists/windows-exporter.yaml](./default-allow-lists/windows-exporter.yaml)                                                                                                       |
 
 ## Metrics tuning and allow lists
 

diff --git a/charts/feature-cluster-metrics/README.md.gotmpl b/charts/feature-cluster-metrics/README.md.gotmpl
@@ -20,30 +20,30 @@ not on the list will be dropped. With [metrics tuning](#metrics-tuning--allow-li
 
 This chart includes the ability to collect metrics from the following:
 
-* The Kubernetes cluster itself
-* Sources like the Kubelet and cAdvisor
-* Common supporting services like [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) and
-[Node Exporter](https://github.com/prometheus/node_exporter)
-* Systems to capture additional data like Kepler
+*   The Kubernetes cluster itself
+*   Sources like the Kubelet and cAdvisor
+*   Common supporting services like [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) and
+    [Node Exporter](https://github.com/prometheus/node_exporter)
+*   Systems to capture additional data like Kepler
 
 ### Metrics sources
 
 The Cluster Metrics feature of the Kubernetes Observability Helm chart includes the following metric systems and
 their default allow lists:
 
-| Metric source | Gathers information about | Allow list |
-| --- | --- | --- |
-| API Server | Kubernetes API Server | NA |
-| [cAdvisor](https://github.com/google/cadvisor)| Containers on each node | [default-allow-lists/cadvisor.yaml](./default-allow-lists/cadvisor.yaml) |
-| [Kepler](https://sustainable-computing.io/) | Kubernetes cluster | [default-allow-lists/kepler.yaml](./default-allow-lists/kepler.yaml) |
-| Kube Controller Manager | Kubernetes Controller Manager | NA |
-| Kube Proxy | Kube Proxy | NA|
-| Kube Scheduler | Kube Scheduler |  NA|
-| Kubelet | Kubernetes information on each node | [default-allow-lists/kubelet.yaml](./default-allow-lists/kubelet.yaml) |
-| [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) |Kubernetes
-resources inside the cluster | [default-allow-lists/kube-state-metrics.yaml](./default-allow-lists/kube-state-metrics.yaml) |
-| [Node Exporter](https://github.com/prometheus/node_exporter) | Linux Kubernetes nodes | [default-allow-lists/node-exporter.yaml](./default-allow-lists/node-exporter.yaml), [default-allow-lists/node-exporter-integration.yaml](./default-allow-lists/node-exporter-integration.yaml)  |
-| [Windows Exporter](https://github.com/prometheus-community/windows_exporter) |  Windows Kubernetes nodes | [default-allow-lists/windows-exporter.yaml](./default-allow-lists/windows-exporter.yaml) |
+| Metric source                                                                | Gathers information about                                                                    | Allow list                                                                                                                                                                                     |
+|------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| API Server                                                                   | Kubernetes API Server                                                                        | NA                                                                                                                                                                                             |
+| [cAdvisor](https://github.com/google/cadvisor)                               | Containers on each node                                                                      | [default-allow-lists/cadvisor.yaml](./default-allow-lists/cadvisor.yaml)                                                                                                                       |
+| [Kepler](https://sustainable-computing.io/)                                  | Kubernetes cluster                                                                           | [default-allow-lists/kepler.yaml](./default-allow-lists/kepler.yaml)                                                                                                                           |
+| Kube Controller Manager                                                      | Kubernetes Controller Manager                                                                | NA                                                                                                                                                                                             |
+| Kube Proxy                                                                   | Kube Proxy                                                                                   | NA                                                                                                                                                                                             |
+| Kube Scheduler                                                               | Kube Scheduler                                                                               | NA                                                                                                                                                                                             |
+| Kubelet                                                                      | Kubernetes information on each node                                                          | [default-allow-lists/kubelet.yaml](./default-allow-lists/kubelet.yaml)                                                                                                                         |
+| [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics)       | Kubernetes                                                                                   |                                                                                                                                                                                                |
+| resources inside the cluster                                                 | [default-allow-lists/kube-state-metrics.yaml](./default-allow-lists/kube-state-metrics.yaml) |                                                                                                                                                                                                |
+| [Node Exporter](https://github.com/prometheus/node_exporter)                 | Linux Kubernetes nodes                                                                       | [default-allow-lists/node-exporter.yaml](./default-allow-lists/node-exporter.yaml), [default-allow-lists/node-exporter-integration.yaml](./default-allow-lists/node-exporter-integration.yaml) |
+| [Windows Exporter](https://github.com/prometheus-community/windows_exporter) | Windows Kubernetes nodes                                                                     | [default-allow-lists/windows-exporter.yaml](./default-allow-lists/windows-exporter.yaml)                                                                                                       |
 
 ## Metrics tuning and allow lists
 

diff --git a/charts/feature-pod-logs/README.md b/charts/feature-pod-logs/README.md
@@ -62,3 +62,9 @@ Be sure perform actual integration testing in a live environment in the main [k8
 | Key | Type | Default | Description |
 |-----|------|---------|-------------|
 | global.platform | string | `""` | The specific platform for this cluster. Will enable compatibility for some platforms. Supported options: (empty) or "openshift". |
+
+### Volume Log Gathering
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| volumeGatherSettings.onlyGatherNewLogLines | bool | `false` | Only gather new log lines since this was deployed. Do not gather historical log lines. |
diff --git a/charts/feature-pod-logs/templates/_volumes.alloy.tpl b/charts/feature-pod-logs/templates/_volumes.alloy.tpl
@@ -30,6 +30,9 @@ local.file_match "pod_logs" {
 
 loki.source.file "pod_logs" {
   targets    = local.file_match.pod_logs.targets
+{{- if .Values.volumeGatherSettings.onlyGatherNewLogLines }}
+  tail_from_end = {{ .Values.volumeGatherSettings.onlyGatherNewLogLines }}
+{{- end }}
   forward_to = [loki.process.pod_logs.receiver]
 }
 {{- end -}}
diff --git a/charts/feature-pod-logs/values.schema.json b/charts/feature-pod-logs/values.schema.json
@@ -53,6 +53,14 @@
         },
         "namespaces": {
             "type": "array"
+        },
+        "volumeGatherSettings": {
+            "type": "object",
+            "properties": {
+                "onlyGatherNewLogLines": {
+                    "type": "boolean"
+                }
+            }
         }
     }
 }
diff --git a/charts/feature-pod-logs/values.yaml b/charts/feature-pod-logs/values.yaml
@@ -40,6 +40,12 @@ labels:
 annotations:
   job: k8s.grafana.com/logs.job
 
+# Settings specific for gathering Pod logs using the "volumes" gather method.
+volumeGatherSettings:
+  # -- Only gather new log lines since this was deployed. Do not gather historical log lines.
+  # @section -- Volume Log Gathering
+  onlyGatherNewLogLines: false
+
 # -- Stage blocks to be added to the loki.process component for pod logs.
 # ([docs](https://grafana.com/docs/alloy/latest/reference/components/loki/loki.process/#blocks))
 # This value is templated so that you can refer to other values from this file.

diff --git a/charts/k8s-monitoring-v1/README.md b/charts/k8s-monitoring-v1/README.md
@@ -481,6 +481,7 @@ The Prometheus and Loki services may be hosted on the same cluster, or remotely
 | logs.pod_logs.labels | object | `{"app_kubernetes_io_name":"app.kubernetes.io/name"}` | Loki labels to set with values copied from the Kubernetes Pod labels. Format: `<loki_label>: <kubernetes_label>`. |
 | logs.pod_logs.namespaces | list | `[]` | Only capture logs from pods in these namespaces (`[]` means all namespaces). |
 | logs.pod_logs.structuredMetadata | object | `{}` | List of labels to turn into structured metadata. If your Loki instance does not support structured metadata, leave this empty. Format: `<structured metadata>: <Loki label>`. |
+| logs.pod_logs.volumeGatherSettings.onlyGatherNewLogLines | bool | `false` | Only gather new log lines since this was deployed. Do not gather historical log lines. |
 
 ### Logs Receiver
 

diff --git a/charts/k8s-monitoring-v1/templates/alloy_config/_pod_logs_discovery.alloy.txt b/charts/k8s-monitoring-v1/templates/alloy_config/_pod_logs_discovery.alloy.txt
@@ -110,6 +110,9 @@ local.file_match "pod_logs" {
 
 loki.source.file "pod_logs" {
   targets    = local.file_match.pod_logs.targets
+{{- if .Values.logs.pod_logs.volumeGatherSettings.onlyGatherNewLogLines }}
+  tail_from_end = {{ .Values.logs.pod_logs.volumeGatherSettings.onlyGatherNewLogLines }}
+{{- end }}
   forward_to = [loki.process.pod_logs.receiver]
 }
 {{ else if eq .Values.logs.pod_logs.gatherMethod "api" }}

diff --git a/charts/k8s-monitoring-v1/values.schema.json b/charts/k8s-monitoring-v1/values.schema.json
@@ -1167,6 +1167,14 @@
                         },
                         "structuredMetadata": {
                             "type": "object"
+                        },
+                        "volumeGatherSettings": {
+                            "type": "object",
+                            "properties": {
+                                "onlyGatherNewLogLines": {
+                                    "type": "boolean"
+                                }
+                            }
                         }
                     }
                 },

diff --git a/charts/k8s-monitoring-v1/values.yaml b/charts/k8s-monitoring-v1/values.yaml
@@ -1707,6 +1707,12 @@ logs:
     # @section -- Logs Scrape: Pod Logs
     structuredMetadata: {}
 
+    # Settings specific for gathering Pod logs using the "volumes" gather method.
+    volumeGatherSettings:
+      # -- Only gather new log lines since this was deployed. Do not gather historical log lines.
+      # @section -- Logs Scrape: Pod Logs
+      onlyGatherNewLogLines: false
+
   # PodLog Objects
   podLogsObjects:
     # -- Enable discovery of Grafana Alloy PodLogs objects.

diff --git a/charts/k8s-monitoring/Chart.lock b/charts/k8s-monitoring/Chart.lock
@@ -45,4 +45,4 @@ dependencies:
   repository: https://grafana.github.io/helm-charts
   version: 0.10.1
 digest: sha256:e2eab36a3b814d7ec09942c3fe46a5c76d606f7c3022628905c41fba7a47b405
-generated: "2024-12-03T20:08:14.380789162Z"
+generated: "2024-12-09T19:06:04.597148-06:00"
diff --git a/charts/k8s-monitoring/charts/k8s-monitoring-feature-annotation-autodiscovery-1.0.0.tgz b/charts/k8s-monitoring/charts/k8s-monitoring-feature-annotation-autodiscovery-1.0.0.tgz
diff --git a/charts/k8s-monitoring/charts/k8s-monitoring-feature-application-observability-1.0.0.tgz b/charts/k8s-monitoring/charts/k8s-monitoring-feature-application-observability-1.0.0.tgz
diff --git a/charts/k8s-monitoring/charts/k8s-monitoring-feature-auto-instrumentation-1.0.0.tgz b/charts/k8s-monitoring/charts/k8s-monitoring-feature-auto-instrumentation-1.0.0.tgz
diff --git a/charts/k8s-monitoring/charts/k8s-monitoring-feature-cluster-events-1.0.0.tgz b/charts/k8s-monitoring/charts/k8s-monitoring-feature-cluster-events-1.0.0.tgz
diff --git a/charts/k8s-monitoring/charts/k8s-monitoring-feature-cluster-metrics-1.0.0.tgz b/charts/k8s-monitoring/charts/k8s-monitoring-feature-cluster-metrics-1.0.0.tgz
diff --git a/charts/k8s-monitoring/charts/k8s-monitoring-feature-integrations-1.0.0.tgz b/charts/k8s-monitoring/charts/k8s-monitoring-feature-integrations-1.0.0.tgz
diff --git a/charts/k8s-monitoring/charts/k8s-monitoring-feature-node-logs-1.0.0.tgz b/charts/k8s-monitoring/charts/k8s-monitoring-feature-node-logs-1.0.0.tgz
diff --git a/charts/k8s-monitoring/charts/k8s-monitoring-feature-pod-logs-1.0.0.tgz b/charts/k8s-monitoring/charts/k8s-monitoring-feature-pod-logs-1.0.0.tgz
diff --git a/charts/k8s-monitoring/charts/k8s-monitoring-feature-profiling-1.0.0.tgz b/charts/k8s-monitoring/charts/k8s-monitoring-feature-profiling-1.0.0.tgz
diff --git a/charts/k8s-monitoring/charts/k8s-monitoring-feature-prometheus-operator-objects-1.0.0.tgz b/charts/k8s-monitoring/charts/k8s-monitoring-feature-prometheus-operator-objects-1.0.0.tgz
diff --git a/charts/k8s-monitoring/tests/integration/auth/configs/nginx-auth-gateway.yaml b/charts/k8s-monitoring/tests/integration/auth/configs/nginx-auth-gateway.yaml
@@ -16,7 +16,7 @@ serverBlock: |-
 
       proxy_pass http://loki.loki.svc:3100/;
     }
-  
+
     location /metrics/bearer/ {
       if ($http_authorization != "Bearer my-bearer-token") {
         return 401;