Difficulty: Moderate
Estimated Time: 10 minutes

Intermediate: (Bonus) Using Prometheus Agent for pure forward-only metric streaming with Thanos Receive

The Thanos project defines a set of components that can be composed together into a highly available metric system with unlimited storage capacity that seamlessly integrates into your existing Prometheus deployments.

But Prometheus project is far from slowing down the development. Together with the community, it constantly evolves to bring value to different network and cluster topologies that we see. Thanos brings distributed and cloud storage and querying to the table, built on the core Prometheus code and design. It allowed Prometheus to focus on critical collection and single cluster monitoring functionalities.

As we learned in the previous tutorial, certain situations require us to collect (pull) data from applications and stream them out of the cluster as soon as possible. Thanos Receive allows doing that by ingesting metrics using the Remote Write protocol that the sender can implement. Typically we recommended using Prometheus with short retention and blocked read and query API as a "lightweight" sender.

In November 2021, however, we, the Prometheus community, introduced a brand new Prometheus mode called "Agent mode". The implementation itself was already battle tested on https://github.com/grafana/agent, where it was available and authored by Robert Fratto since 2020.

The agent mode is optimized for efficient metric scraping and forwarding (i.e. immediate metric removal once it's securely delivered to a remote location). Since this is incredibly helpful for the Thanos community, we wanted to give you first-hand experience deploying Prometheus Agent together with Thanos Receive in this course.

In this tutorial, you will learn:

  • How to reduce Prometheus based client-side metric collection to a minimum, using the new Prometheus "Agent mode" with Thanos Receiver, explained in the previous tutorial.

NOTE: This course uses docker containers with pre-built Thanos, Prometheus, and Minio Docker images available publicly.

Prerequisites

Please complete tutorial #3 first: Intermediate: Streaming metrics from remote source with Thanos Receive 🤗

Feedback

Do you see any bug, typo in the tutorial, or do you have some feedback for us? Let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io

Contributed by:

Summary

Congratulations! 🎉🎉🎉

Batmobile is now well-monitored! Batman says thank you!

batman

You completed this Prometheus Agent tutorial. Let's summarize what we learned:

  • Thanos Receive is a component that implements the Prometheus Remote Write protocol.
  • Prometheus Agent can be deployed to remote write its metric data in real-time to another server that implements the Remote Write protocol.
  • Prometheus Agent uses a fraction of the resources as normal Prometheus, by not enabling features like: local metrics storage and local query APIs.

See next courses for other tutorials about different deployment models and more advanced features of Thanos!

Further Reading

Feedback

Do you see any bug, typo in the tutorial or you have some feedback for us?

let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io

Bonus: Using Prometheus Agent for streaming metrics to Thanos Receive

Step 1 of 3

Problem Statement & Setup

Problem Statement

Let's get back to our example from Tutorial 3. Imagine you run a company called Wayne Enterprises. In tutorial 3, we established monitoring of two special clusters: Batcave & Batcomputer. These are special because they do not expose public endpoints to the Prometheus instances running there for security reasons, so we used the Remote Write protocol to stream all metrics to Thanos Receive in our centralized space.

Let's imagine we want to expand our Wayne Enterprises by adding metrics collection to applications running on smaller devices inside more mobile Batman tools: Batmobile and Batcopter.

Each of these vehicles has a smaller computer that runs applications from which we want to scrape Prometheus-like metrics using OpenMetrics format.

As the person responsible for implementing monitoring in these environments, you have three requirements to meet:

  1. Implement a global view of this data. Wayne Enterprises needs to know what is happening in all company parts - including secret ones!
  2. Batmobile and Batcopter can be out of network for some duration of time. You don't want to lose precious data.
  3. Batmobile and Batcopter environments are very resource constrained, so you want an efficient solution that avoids extra computations and storage.

Firstly, let us set up Thanos as we explained in the previous tutorial.

Setup Central Platform

As you might remember from the previous tutorial, in the simplest form for streaming use cases, we need to deploy Thanos Receive and Thanos Querier.

Let's run Thanos Receive:

docker run -d --rm \
    -v $(pwd)/receive-data:/receive/data \
    --net=host \
    --name receive \
    quay.io/thanos/thanos:v0.21.0 \
    receive \
    --tsdb.path "/receive/data" \
    --grpc-address 127.0.0.1:10907 \
    --http-address 127.0.0.1:10909 \
    --label "receive_replica=\"0\"" \
    --label "receive_cluster=\"wayne-enterprises\"" \
    --remote-write.address 127.0.0.1:10908

This starts Thanos Receive that listens on http://127.0.0.1:10908/api/v1/receive endpoint for Remote Write and on 127.0.0.1:10907 for Thanos StoreAPI.

Next, let us run a Thanos Query instance connected to Thanos Receive:

docker run -d --rm \
--net=host \
--name query \
quay.io/thanos/thanos:v0.21.0 \
query \
--http-address "0.0.0.0:39090" \
--store "127.0.0.1:10907"

Verify that Thanos Query is working and configured correctly by looking at the 'stores' tab here (try refreshing the Thanos UI if it does not show up straight away).

We should see Receive store on this page and, as expected, no metric data since we did not connect any Remote Write sender yet.

With our "central" platform that is ready to ingest metrics, we can now start to architect our collection pipeline that will stream all metrics to our Wayne Enterprises.

prom-agent-batmobile.yaml
prom-agent-batcopter.yaml
Terminal
Prometheus Agent Batmobile
Prometheus Agent Batcopter
Thanos Query