Custom metrics Adapter

This adapter contains an implementation of external metrics API. It is therefore suitable for use with the autoscaling/v2 Horizontal Pod Autoscaler in Kubernetes 1.9+.

Use kustomize to customise your deployment

  1. Clone the source code:
git clone git@github.com:apache/skywalking-swck.git

  1. Edit file adapter/config/adapter/kustomization.yaml file to change your preferences. If you prefer to your private docker image, a quick path to override ADAPTER_IMG environment variable : export ADAPTER_IMG=<private registry>/metrics-adapter:<tag>

  2. Use make to generate the final manifests and deploy:

make -C adapter deploy

Configuration

The adapter takes the standard Kubernetes generic API server arguments (including those for authentication and authorization). By default, it will attempt to using Kubernetes in-cluster config to connect to the cluster.

It takes the following addition arguments specific to configuring how the adapter talks to SkyWalking OAP cluster:

  • --oap-addr The address of OAP cluster.
  • --metric-filter-regex A regular expression to filter metrics retrieved from OAP cluster.
  • --refresh-interval This is the interval at which to update the cache of available metrics from OAP cluster.
  • --namespace A prefix to which metrics are appended. The format is ‘namespace|metric_name’, defaults to skywalking.apache.org

HPA Configuration

External metrics allow you to autoscale your cluster based on any metric available in OAP cluster. Just provide a metric block with a name and selector, and use the External metric type.

- type: External
  external:
    metric:
      name: <metric_name>
      selector:
        matchLabels:
          <label_key>: <label_value>
          ...
    target:
      ....
  • metric_name: The name of metric generated by OAL or other subsystem.
  • label: label_key is the entity name of skywalking metrics. if the label value contains special characters more than ., - and _, service.str.<number> represent the literal of label value, and service.byte.<number> could encode these special characters to hex bytes.

Supposing the service name is v1|productpage|bookinfo|demo, the matchLabels should be like the below piece:

matchLabels:
  "service.str.0":  "v1"
  "service.byte.1": "7c" // the hex byte of "|"
  "service.str.2":  "productpage"
  "service.byte.3": "7c"
  "service.str.4":  "bookinfo"
  "service.byte.5": "7c"
  "service.str.6":  "demo"

Caveats: byte label only accept a single character. That means || should be transformed to service.byte.0:"7c" and service.byte.1:"7c" instead of service.byte.0:"7c7c"

The options of label keys are:

  • service, service.str.<number> or service.byte.<number> The name of the service.
  • instance, instance.str.<number> or instance.byte.<number> The name of the service instance.
  • endpoint, endpoint.str.<number> or endpoint.byte.<number> The name of the endpoint.
  • label, label.str.<number> or label.byte.<number> is optional, The labels you need to query, used for querying multi-labels metrics. Unlike swctl, this key only supports a single label due to the specification of the custom metrics API.

For example, if your application name is front_gateway, you could add the following section to your HorizontalPodAutoscaler manifest to specify that you need less than 80ms of 90th latency.

- type: External
  external:
    metric:
      name: skywalking.apache.org|service_percentile
      selector:
        matchLabels:
           service: front_gateway
            # The index of [P50, P75, P90, P95, P99]. 2 is the index of P90(90%)
           label: "2"
    target:
      type: Value
      value: 80

If the service is v1|productpage|bookinfo|demo|-:

- type: External
  external:
    metric:
      name: skywalking.apache.org|service_cpm
      selector:
        matchLabels:
          "service.str.0":  "v1"
          "service.byte.1": "7c"
          "service.str.2":  "productpage"
          "service.byte.3": "7c"
          "service.str.4":  "bookinfo"
          "service.byte.5": "7c"
          "service.str.6":  "demo"
          "service.byte.7": "7c"
          "service.byte.8": "2d"
    target:
      type: Value
      value: 80