Kubernetes (K8s) monitoring

SkyWalking leverages K8s kube-state-metrics (KSM) and cAdvisor for collecting metrics data from K8s. It leverages OpenTelemetry Collector to transfer the metrics to OpenTelemetry receiver and into the Meter System. This feature requires authorizing the OAP Server to access K8s’s API Server.

Data flow

  1. K8s kube-state-metrics and cAdvisor collect metrics data from K8s.
  2. OpenTelemetry Collector fetches metrics from kube-state-metrics and cAdvisor via Prometheus Receiver and pushes metrics to SkyWalking OAP Server via the OpenCensus gRPC Exporter or OpenTelemetry gRPC exporter.
  3. The SkyWalking OAP Server access to K8s’s API Server gets meta info and parses the expression with MAL to filter/calculate/aggregate and store the results.

Setup

  1. Setup kube-state-metric.
  2. cAdvisor is integrated into kubelet by default.
  3. Set up OpenTelemetry Collector . For details on Prometheus Receiver in OpenTelemetry Collector for K8s, refer to here. For a quick start, we have provided a complete example of configuration and recommended version; you can refer to showcase.
  4. Config SkyWalking OpenTelemetry receiver.

Kubernetes Cluster Monitoring

K8s cluster monitoring provides monitoring of the status and resources of the whole cluster and each node. K8s cluster as a Service in OAP, K8s node as an Instance in OAP, and land on the Layer: K8S.

Kubernetes Cluster Supported Metrics

Monitoring Panel Unit Metric Name Description Data Source
Node Total k8s_cluster_node_total The number of nodes K8s kube-state-metrics
Namespace Total k8s_cluster_namespace_total The number of namespaces K8s kube-state-metrics
Deployment Total k8s_cluster_deployment_total The number of deployments K8s kube-state-metrics
Service Total k8s_cluster_service_total The number of services K8s kube-state-metrics
Pod Total k8s_cluster_pod_total The number of pods K8s kube-state-metrics
Container Total k8s_cluster_container_total The number of containers K8s kube-state-metrics
CPU Resources m k8s_cluster_cpu_cores
k8s_cluster_cpu_cores_requests
k8s_cluster_cpu_cores_limits
k8s_cluster_cpu_cores_allocatable		 The capacity and the Requests / Limits / Allocatable of the CPU K8s kube-state-metrics
Memory Resources Gi k8s_cluster_memory_total
k8s_cluster_memory_requests
k8s_cluster_memory_limits
k8s_cluster_memory_allocatable		 The capacity and the Requests / Limits / Allocatable of the memory K8s kube-state-metrics
Storage Resources Gi k8s_cluster_storage_total
k8s_cluster_storage_allocatable		 The capacity and allocatable of the storage K8s kube-state-metrics
Node Status k8s_cluster_node_status The current status of the nodes K8s kube-state-metrics
Deployment Status k8s_cluster_deployment_status The current status of the deployment K8s kube-state-metrics
Deployment Spec Replicas k8s_cluster_deployment_spec_replicas The number of desired pods for a deployment K8s kube-state-metrics
Service Status k8s_cluster_service_pod_status The services current status, depending on the related pods' status K8s kube-state-metrics
Pod Status Not Running k8s_cluster_pod_status_not_running The pods which are not running in the current phase K8s kube-state-metrics
Pod Status Waiting k8s_cluster_pod_status_waiting The pods and containers which are currently in the waiting status, with reasons shown K8s kube-state-metrics
Pod Status Terminated k8s_cluster_container_status_terminated The pods and containers which are currently in the terminated status, with reasons shown K8s kube-state-metrics

Kubernetes Cluster Node Supported Metrics

Monitoring Panel Unit Metric Name Description Data Source
Pod Total k8s_node_pod_total The number of pods in this node K8s kube-state-metrics
Node Status k8s_node_node_status The current status of this node K8s kube-state-metrics
CPU Resources m k8s_node_cpu_cores
k8s_node_cpu_cores_allocatable
k8s_node_cpu_cores_requests
k8s_node_cpu_cores_limits		 The capacity and the requests / Limits / Allocatable of the CPU K8s kube-state-metrics
Memory Resources Gi k8s_node_memory_total
k8s_node_memory_allocatable
k8s_node_memory_requests
k8s_node_memory_limits		 The capacity and the requests / Limits / Allocatable of the memory K8s kube-state-metrics
Storage Resources Gi k8s_node_storage_total
k8s_node_storage_allocatable		 The capacity and allocatable of the storage K8s kube-state-metrics
CPU Usage m k8s_node_cpu_usage The total usage of the CPU core, if there are 2 cores the maximum usage is 2000m cAdvisor
Memory Usage Gi k8s_node_memory_usage The totaly memory usage cAdvisor
Network I/O KB/s k8s_node_network_receive
k8s_node_network_transmit		 The network receive and transmit cAdvisor

Kubernetes Service Monitoring

K8s Service Monitoring provides observabilities into service status and resources from Kubernetes. K8s Service as a Service in OAP and land on the Layer: K8S_SERVICE.

Kubernetes Service Supported Metrics

Monitoring Panel Unit Metric Name Description Data Source
Service Pod Total k8s_service_pod_total The number of pods K8s kube-state-metrics
Service Pod Status k8s_service_pod_status The current status of pods K8s kube-state-metrics
Service CPU Resources m k8s_service_cpu_cores_requests
k8s_service_cpu_cores_limits		 The CPU resources requests / Limits of this service K8s kube-state-metrics
Service Memory Resources MB k8s_service_memory_requests
k8s_service_memory_limits		 The memory resources requests / Limits of this service K8s kube-state-metrics
Pod CPU Usage m k8s_service_pod_cpu_usage The CPU resources total usage of pods cAdvisor
Pod Memory Usage MB k8s_service_pod_memory_usage The memory resources total usage of pods cAdvisor
Pod Waiting k8s_service_pod_status_waiting The pods and containers which are currently in the waiting status, with reasons shown K8s kube-state-metrics
Pod Terminated k8s_service_pod_status_terminated The pods and containers which are currently in the terminated status, with reasons shown K8s kube-state-metrics
Pod Restarts k8s_service_pod_status_restarts_total The number of per container restarts related to the pods K8s kube-state-metrics

Customizations

You can customize your own metrics/expression/dashboard panel. The metrics definition and expression rules are found in /config/otel-rules/k8s/k8s-cluster.yaml，/config/otel-rules/k8s/k8s-node.yaml, /config/otel-rules/k8s/k8s-service.yaml. The K8s Cluster dashboard panel configurations are found in /config/ui-initialized-templates/k8s. The K8s Service dashboard panel configurations are found in /config/ui-initialized-templates/k8s_service.