Prometheus is an open-source monitoring solution that can be integrated with Envoy's own stats.

Now, let's see the prometheus configuration:

prometheus.yml

We can see that the prometheus config is using a job named envoy.

scrape_configs:
  - job_name: 'envoy'
    metrics_path: /stats/prometheus

Start prometheus with command:

docker run -d -p 9090:9090 \
    -v /root/envoy/prometheus.yml:/etc/prometheus/prometheus.yml \
    --name prometheus-server \
    prom/prometheus

You can access to the prometheus dashboard using this link:

https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com/targets

Distributed tracing allows developers to obtain visualizations of call flows in large service oriented architectures. It can be invaluable in understanding serialization, parallelism, and sources of latency. For this example we will use Jaeger, an open source, end-to-end distributed tracing.

We can see the tracing configuration in envoy.yaml

tracing:
  http:
    name: envoy.zipkin
    config:
      collector_cluster: 172.18.0.6
      collector_endpoint: "/api/v1/spans"
      shared_span_context: false

Ensure that Jaeger is configured to accept Zipkin requests via the COLLECTOR_ZIPKIN_HTTP_PORT Environment Variable.

One important configuration for our example is telling to the connection manager that generates the x-request-id header if it does not exist.

 generate_request_id: true
 tracing:
   operation_name: egress
 

docker run -d --name jaeger -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 -p 9411:9411 -p 5775:5775/udp -p 16686:16686 jaegertracing/all-in-one:latest

Now, we should generate traffic performing requests to the envoy proxy using this command:

while true; do curl localhost; sleep .5; done

You can see the statistics generated in plan text in this URL:

https://[[HOST_SUBDOMAIN]]-9901-[[KATACODA_HOST]].environments.katacoda.com/stats/prometheus

And you can use one particular field to build a graph, for example: envoy_cluster_external_upstream_rq

To build the graph, go to the dashboard: https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com/graph

And use this query: envoy_cluster_external_upstream_rq{envoy_cluster_name="targetCluster"}

Now, let's run grafana with this command:

docker run --name=grafana -d -p 3000:3000 grafana/grafana

And access to the dashboard using this url:

https://[[HOST_SUBDOMAIN]]-3000-[[KATACODA_HOST]].environments.katacoda.com/

And use this default credentials: admin \ admin

The initial step will ask you to change the password, you can do it if you want or you can skip this step.

The first step to create a dashboard is to have a datasource. Let's define a datasource with phometheus data configured before:

The URL for phometheus should be:

http://172.18.0.5:9090

For all the other fields you can use the default values

Test and save your datasource.

Build a dashboard

In order to build a dashboard you can build one from scratch or you can use an existing one, for example: https://grafana.com/dashboards/6693

Let's use this existing dashboard. Copy the ID, and use the option Import.

6693

Select Prometheus as the data source and Import.

View Dashboard for the targetCluster

Change Traffic

while true; do curl localhost; sleep .1; done

Should see a big spike in the traffic on the dashboard.

Generate Errors

curl 172.18.0.3/unhealthy; curl 172.18.0.4/unhealthy;

Within Prometheus, you will also see a increase in the number of 500 error messages.

https://[[HOST_SUBDOMAIN]]-9090-[[KATACODA_HOST]].environments.katacoda.com/graph

envoy_cluster_external_upstream_rq{envoy_cluster_name="targetCluster"}

You can see the traces in Jaeger using this URL https://[[HOST_SUBDOMAIN]]-16686-[[KATACODA_HOST]].environments.katacoda.com/search