Observe Service Mesh through ALS

Envoy Access Log Service (ALS) provides full logs on routed RPC, including HTTP and TCP.

Background

The solution was initialized and first implemented by Sheng Wu, Hongtao Gao, Lizan Zhou, and Dhi Aurrahman on May 17, 2019, and was presented at KubeCon China 2019. Here is a video recording of the presentation.

SkyWalking is the first open source project that introduced an ALS-based solution to the world. This solution provides a new take on observability with a lightweight payload on the service mesh.

Enable ALS and SkyWalking Receiver

You need the following steps to set up ALS.

Enable envoyAccessLogService in ProxyConfig and set the ALS address to where the SkyWalking OAP listens. In Istio version 1.6.0+, if Istio is installed with demo profile, you can enable ALS with this command:
```
istioctl manifest apply \
  --set profile=demo \
  --set meshConfig.enableEnvoyAccessLogService=true \
  --set meshConfig.defaultConfig.envoyAccessLogService.address=<skywalking-oap.skywalking.svc:11800>
```
Note: Replace <skywalking-oap.skywalking.svc:11800> with the real address where SkyWalking OAP is deployed.
Activate SkyWalking Envoy Receiver. (activated in default)

envoy-metric:
  selector: ${SW_ENVOY_METRIC:default}

Choose an ALS analyzer. There are two available analyzers for both HTTP access logs and TCP access logs: k8s-mesh and mx-mesh. Set the system environment variables SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS and SW_ENVOY_METRIC_ALS_TCP_ANALYSIS, such as SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS=mx-mesh and SW_ENVOY_METRIC_ALS_TCP_ANALYSIS=mx-mesh, or in application.yaml to activate the analyzers. For more about the analyzers, see SkyWalking ALS Analyzers.
```
envoy-metric:
  selector: ${SW_ENVOY_METRIC:default}
  default:
    acceptMetricsService: ${SW_ENVOY_METRIC_SERVICE:true}
    alsHTTPAnalysis: ${SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS:""} # Setting the system env variable would override this. 
    alsTCPAnalysis: ${SW_ENVOY_METRIC_ALS_TCP_ANALYSIS:""}
```
To use multiple analyzers as a fallback, please use , to concatenate.

Example

Here’s an example on installing Istio and deploying SkyWalking by Helm chart.

istioctl install \
  --set profile=demo \
  --set meshConfig.enableEnvoyAccessLogService=true \
  --set meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-oap.istio-system:11800

git clone https://github.com/apache/skywalking-kubernetes.git
cd skywalking-kubernetes/chart

helm repo add elastic https://helm.elastic.co

helm dep up skywalking

helm install 8.1.0 skywalking -n istio-system \
  --set oap.env.SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS=mx-mesh \
  --set oap.env.SW_ENVOY_METRIC_ALS_TCP_ANALYSIS=mx-mesh \
  --set fullnameOverride=skywalking \
  --set oap.envoy.als.enabled=true

You can use kubectl -n istio-system logs -l app=skywalking | grep "K8sALSServiceMeshHTTPAnalysis" to ensure that OAP ALS mx-mesh analyzer has been activated.

SkyWalking ALS Analyzers

There are several available analyzers: k8s-mesh, mx-mesh, and persistence. You can specify one or more analyzers to analyze the access logs. When multiple analyzers are specified, it acts as a fast-success mechanism: SkyWalking loops over the analyzers and use them to analyze the logs. Once there is an analyzer that is able to produce a result, it stops the loop.

`k8s-mesh`

k8s-mesh uses the metadata from Kubernetes cluster, hence in this analyzer OAP needs access roles to Pod, Service, and Endpoints.

The blog illustrates the details of how it works, and a step-by-step tutorial to apply it into the bookinfo application.

`mx-mesh`

mx-mesh uses the Envoy metadata exchange mechanism to get the service name, etc. This analyzer requires Istio to enable the metadata exchange plugin (you can enable it by --set values.telemetry.v2.enabled=true, or if you’re using Istio 1.7+ and installing it with profile demo/preview, it should already be enabled).

The blog illustrates the details of how it works, and a step-by-step tutorial to apply it into the Online Boutique system.

`persistence`

persistence analyzer adapts the Envoy access log format to SkyWalking’s native log format, and forwards the formatted logs to LAL, where you can configure persistent conditions, such as sampler, only persist error logs, etc. SkyWalking provides a default configuration file envoy-als.yaml that you can adjust as per your needs. Please make sure to activate this rule via adding the rule name envoy-als into config item log-analyzer/default/lalFiles (or environment variable SW_LOG_LAL_FILES, e.g. SW_LOG_LAL_FILES=envoy-als).

Attention: Since the persistence analyzer also needs a mechanism to map the logs into responding services, you need to configure at least one of k8s-mesh or mx-mesh as its antecedent so that persistence analyzer knows which service the logs belong to. For example, you should set envoy-metric/default/alsHTTPAnalysis (or environment variable SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS) to something like k8s-mesh,persistence, mx-mesh,persistence, or mx-mesh,k8s-mesh,persistence.