Collecting Metrics

This task shows how to configure Istio to automatically gather telemetry forservices in a mesh. At the end of this task, a new metric will be enabled forcalls to services within your mesh.

The Bookinfo sample application is usedas the example application throughout this task.

Before you begin

• Install Istio in your cluster and deploy anapplication. This task assumes that Mixer is setup in a default configuration(—configDefaultNamespace=istio-system). If you use a differentvalue, update the configuration and commands in this task to match the value.

Collecting new metrics

• Apply a YAML file with configuration for the new metricthat Istio will generate and collect automatically.

Zip

$ kubectl apply -f @samples/bookinfo/telemetry/metrics.yaml@

If you use Istio 1.1.2 or prior, please use the following configuration instead:

Zip

$ kubectl apply -f @samples/bookinfo/telemetry/metrics-crd.yaml@

• Send traffic to the sample application.

For the Bookinfo sample, visit http://$GATEWAY_URL/productpage in your webbrowser or issue the following command:

$ curl http://$GATEWAY_URL/productpage

• Verify that the new metric values are being generated and collected.

In a Kubernetes environment, setup port-forwarding for Prometheus byexecuting the following command:

$ kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=prometheus -o jsonpath='{.items[0].metadata.name}') 9090:9090 &

View values for the new metric in the Prometheus browser window. Select Graph.Enter the istio_double_request_count metric and select Execute.The table displayed in theConsole tab includes entries similar to:

istio_double_request_count{destination="details-v1",instance="172.17.0.12:42422",job="istio-mesh",message="twice the fun!",reporter="client",source="productpage-v1"}   8
istio_double_request_count{destination="details-v1",instance="172.17.0.12:42422",job="istio-mesh",message="twice the fun!",reporter="server",source="productpage-v1"}   8
istio_double_request_count{destination="istio-policy",instance="172.17.0.12:42422",job="istio-mesh",message="twice the fun!",reporter="server",source="details-v1"}   4
istio_double_request_count{destination="istio-policy",instance="172.17.0.12:42422",job="istio-mesh",message="twice the fun!",reporter="server",source="istio-ingressgateway"}   4

For more on querying Prometheus for metric values, see theQuerying Istio Metrics task.

Understanding the metrics configuration

In this task, you added Istio configuration that instructed Mixer toautomatically generate and report a new metric for alltraffic within the mesh.

The added configuration controlled three pieces of Mixer functionality:

• Generation of instances (in this example, metric values)from Istio attributes

• Creation of handlers (configured Mixer adapters) capable of processinggenerated instances

• Dispatch of instances to handlers according to a set of rules

The metrics configuration directs Mixer to send metric values to Prometheus. Ituses three stanzas (or blocks) of configuration: instance configuration,handler configuration, and rule configuration.

The kind: instance stanza of configuration defines a schema for generated metric values(or instances) for a new metric named doublerequestcount. This instanceconfiguration tells Mixer how to generate metric values for any given request,based on the attributes reported by Envoy (and generated by Mixer itself).

For each instance of doublerequestcount, the configuration directs Mixer tosupply a value of 2 for the instance. Because Istio generates an instance foreach request, this means that this metric records a value equal to twice thetotal number of requests received.

A set of dimensions are specified for each doublerequestcountinstance. Dimensions provide a way to slice, aggregate, and analyze metric dataaccording to different needs and directions of inquiry. For instance, it may bedesirable to only consider requests for a certain destination service whentroubleshooting application behavior.

The configuration instructs Mixer to populate values for these dimensions basedon attribute values and literal values. For instance, for the sourcedimension, the new configuration requests that the value be taken from thesource.workload.name attribute. If that attribute value is not populated, the ruleinstructs Mixer to use a default value of "unknown". For the messagedimension, a literal value of "twice the fun!" will be used for all instances.

The kind: handler stanza of configuration defines a handler nameddoublehandler. The handler spec configures how the Prometheus adapter codetranslates received metric instances into Prometheus-formatted values that canbe processed by a Prometheus backend. This configuration specified a newPrometheus metric named doublerequest_count. The Prometheus adapter prependsthe istio namespace to all metric names, therefore this metric will show upin Prometheus as istio_double_request_count. The metric has three labelsmatching the dimensions configured for doublerequestcount instances.

Mixer instances are matched to Prometheus metrics via the instance_name parameter.The instance_name values must be the fully-qualified name for Mixer instances (example:doublerequestcount.instance.istio-system).

The kind: rule stanza of configuration defines a new rule named doubleprom. Therule directs Mixer to send all doublerequestcount instances to thedoublehandler handler. Because there is no match clause in therule, and because the rule is in the configured default configuration namespace(istio-system), the rule is executed for all requests in the mesh.

Cleanup

• Remove the new metrics configuration:

Zip

$ kubectl delete -f @samples/bookinfo/telemetry/metrics.yaml@

If you are using Istio 1.1.2 or prior:

Zip

$ kubectl delete -f @samples/bookinfo/telemetry/metrics-crd.yaml@

• Remove any kubectl port-forward processes that may still be running:

$ killall kubectl

• If you are not planning to explore any follow-on tasks, refer to theBookinfo cleanup instructionsto shutdown the application.

See also

Collecting Metrics for TCP services

This task shows you how to configure Istio to collect metrics for TCP services.

Querying Metrics from Prometheus

This task shows you how to query for Istio Metrics using Prometheus.

Mixer and the SPOF Myth

Improving availability and reducing latency.

Mixer Adapter Model

Provides an overview of Mixer's plug-in architecture.

Collecting Logs

This task shows you how to configure Istio to collect and customize logs.

Getting Envoy's Access Logs

This task shows you how to configure Envoy proxies to print access log to their standard output.