# Prometheus

Prometheus is an open-source system for collecting time-series metrics, querying them with PromQL and alerting. Learn scraping, labels, exporters and alerts.

## Overview

Prometheus is an open-source monitoring system and time-series database. It collects numeric measurements, called metrics, from applications and infrastructure at regular intervals, stores them with timestamps, lets you query them with a language called PromQL, and fires alerts when a rule says something is wrong. It is the most common way to monitor services running on [Kubernetes](/tools/kubernetes).

An everyday analogy is a nurse doing ward rounds. Every few minutes the nurse visits each bed and writes down pulse, temperature and blood pressure on a chart. Patients do not have to call anyone; the nurse comes to them. If a reading crosses a line, the nurse calls the doctor. Prometheus does the same: it visits each target on a schedule, records the numbers, and raises an alarm when a rule is broken.

For AI systems, Prometheus is how teams see request rates, latency percentiles, error counts, GPU utilisation, memory use and queue depth for model servers such as [vLLM](/tools/vllm) or [Triton](/tools/triton), many of which expose Prometheus metrics out of the box. It is usually paired with [Grafana](/tools/grafana) for dashboards.

## What it is

The Prometheus ecosystem has a few clear parts.

| Component | Role |
|---|---|
| Prometheus server | Discovers targets, scrapes metrics over HTTP, stores them in its local time-series database, evaluates rules and answers queries |
| Client libraries | Let application code define and expose metrics (Go, Java, Python, Ruby, Rust and others) |
| Exporters | Translate metrics from systems that do not speak Prometheus, such as the node exporter for Linux hosts |
| Pushgateway | Holds metrics pushed by short-lived batch jobs so the server can scrape them |
| Alertmanager | Receives alerts from Prometheus, groups, deduplicates and silences them, and routes notifications |
| Service discovery | Finds targets automatically from Kubernetes, cloud APIs, DNS or files |
| PromQL | The query language for selecting and aggregating time series |

The data model is simple. Each time series is identified by a metric name and a set of labels, which are key-value pairs. For example, `http_requests_total{service="scoring", status="500"}` is one series. Each series holds a stream of samples: a timestamp and a floating-point value.

## Where it came from

Prometheus was started at SoundCloud in 2012, inspired by Google's internal Borgmon monitoring system. It was developed in the open and adopted by many organisations. In 2016 it joined the Cloud Native Computing Foundation (CNCF) as the foundation's second hosted project after Kubernetes, and in August 2018 it became the second project to graduate.

Prometheus 2.0, released in 2017, introduced a new storage engine. Prometheus 3.0, the first major release in seven years, became generally available on 14 November 2024. It brought a new default web interface, Remote Write 2.0, UTF-8 support for metric and label names, and the ability to receive OpenTelemetry metrics natively over OTLP, with native histograms available as a feature. The project is maintained independently of any single company.

## How it works

### 1. Instrument and expose

An application uses a client library to define metrics and serves them at an HTTP endpoint, usually `/metrics`, in a plain text format. For software you cannot change, an exporter does this on its behalf.

### 2. Discover targets

Prometheus reads its configuration and service discovery to build a list of targets. In Kubernetes it can find every pod with a given annotation or every endpoint behind a service.

### 3. Scrape

On a fixed interval, often 15 or 30 seconds, Prometheus sends an HTTP request to each target and parses the response. This is the pull model: the server fetches, rather than applications pushing. A synthetic `up` metric records whether each scrape succeeded, which is itself a useful health signal.

### 4. Store

Samples go into the local time-series database. Recent data sits in memory and a write-ahead log, then is compacted into blocks on disk. Retention is configurable. For long-term or global storage, Prometheus can forward data with remote write to systems such as Thanos, Cortex or Mimir.

### 5. Evaluate rules

Recording rules precompute expensive queries into new series. Alerting rules are PromQL expressions with a duration; if an expression stays true for that long, the alert fires.

### 6. Alert and query

Firing alerts go to Alertmanager, which routes them to chat, email or paging tools. Dashboards and people query the server with PromQL.

A typical alert rule looks like this:

```yaml
groups:
- name: model-serving
  rules:
  - alert: HighErrorRate
    expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.02
    for: 10m
    labels:
      severity: page
```

## Key concepts and terms

| Term | Meaning |
|---|---|
| Metric | A named measurement, such as request count or memory in bytes. |
| Label | A key-value pair that distinguishes series, such as `service` or `region`. |
| Time series | One metric name plus one exact label set, with its samples over time. |
| Sample | A single timestamped value. |
| Counter | A value that only goes up, reset on restart; used with `rate()`. |
| Gauge | A value that can go up and down, such as temperature or queue length. |
| Histogram | Counts of observations in buckets, used to estimate percentiles. |
| Scrape | One HTTP fetch of a target's metrics. |
| Exporter | A bridge exposing another system's metrics in Prometheus format. |
| Cardinality | The number of distinct series; grows with label combinations. |
| Recording rule | A rule that saves a query's result as a new series. |
| Remote write | Forwarding samples to an external storage system. |

## A worked example

This scenario is illustrative. A team serves a text-classification model behind an HTTP API on four Kubernetes pods. Users complain of occasional slowness, but nobody can say how often.

The team adds a Python client library to the service and defines three metrics: a counter `inference_requests_total` with labels `model` and `status`, a histogram `inference_latency_seconds` with buckets from 0.01 to 5 seconds, and a gauge `batch_queue_size`. They also deploy the NVIDIA DCGM exporter to expose GPU utilisation and memory. Prometheus scrapes every 15 seconds.

A PromQL query, `histogram_quantile(0.99, sum by (le) (rate(inference_latency_seconds_bucket[5m])))`, shows that the 99th percentile latency is normally 180 ms but jumps to 2.4 seconds roughly twice an hour. Overlaying `batch_queue_size` shows the spikes coincide with the queue passing 200 requests, while GPU utilisation stays at 60%. The bottleneck is batching configuration, not GPU capacity. After tuning, p99 stays below 300 ms. The team adds an alert that pages if p99 exceeds 500 ms for ten minutes, and a recording rule so the dashboard query stays cheap.

## Strengths and limitations

**Strengths:**

- Simple, powerful data model with labels and a capable query language.
- Single binary with local storage; no external database required.
- Pull model makes target health visible and configuration central.
- Excellent Kubernetes integration and a huge range of exporters.
- Open standard exposition format widely supported by other software.
- Native OTLP ingestion since 3.0 eases mixing with [OpenTelemetry](/tools/opentelemetry).

**Limitations:**

- A single server is not clustered; high availability and long retention need extra systems.
- High-cardinality labels, such as user IDs, can exhaust memory.
- Metrics only: no logs or traces, which need other tools.
- Pull is awkward for short-lived jobs and some network setups.
- Not designed for exact per-event accounting such as billing.

## How it compares with alternatives

| Tool | Approach | When it fits better |
|---|---|---|
| Grafana Mimir, Thanos, Cortex | Horizontally scalable, long-term storage compatible with Prometheus | Many clusters, long retention, global queries |
| VictoriaMetrics | Prometheus-compatible time-series database focused on efficiency | Large volumes with lower resource use |
| InfluxDB | Push-based time-series database with its own query languages | IoT and event-style data pushed from devices |
| Managed cloud monitoring (CloudWatch and similar) | Hosted metrics tied to one cloud | Teams that want no self-hosting and live in one cloud |
| [OpenTelemetry](/tools/opentelemetry) Collector | Vendor-neutral pipeline for metrics, logs and traces | Collection layer feeding Prometheus or other back ends |

Prometheus is often the collection and alerting core, with a scalable store added for retention and Grafana on top. See [model monitoring](/tools/model-monitoring) for the model-quality side.

## Practical guidance and best practices

1. Follow naming conventions: base units such as seconds and bytes, and `_total` for counters.
2. Keep label cardinality bounded; never use user IDs, request IDs or raw URLs as labels.
3. Use histograms for latency and compute percentiles with `histogram_quantile`, rather than averaging.
4. Alert on symptoms users feel, such as error rate and latency, rather than every internal cause.
5. Give every alert a `for` duration and a runbook link to avoid noisy, unactionable pages.
6. Use recording rules for dashboard queries that run often over large ranges.
7. Run two identical Prometheus servers for high availability and let Alertmanager deduplicate.
8. Add remote write to a long-term store if you need more than a few weeks of history.
9. Follow the migration guide when moving from 2.x to 3.x, upgrading to 2.55 first as the project recommends.

## Common mistakes

- Putting unbounded values in labels and crashing the server with millions of series.
- Averaging pre-computed percentiles across pods, which gives meaningless numbers.
- Using `rate()` on gauges, or reading counters directly without `rate()`.
- Alerting on every CPU spike, so people learn to ignore pages.
- Treating local Prometheus storage as a permanent archive without backups or remote write.
- Forgetting to monitor Prometheus itself, including failed scrapes and rule evaluation errors.

## Frequently asked questions

### Does Prometheus push or pull?

It pulls. The server scrapes targets over HTTP on a schedule. Short-lived jobs can push to the Pushgateway, and Prometheus 3.0 can also receive OTLP metrics pushed to it.

### Can Prometheus store data for years?

Locally it can keep data as long as disk allows, but it is not designed as a durable, replicated archive. For long retention, forward data to Thanos, Mimir, Cortex or a similar system.

### What is PromQL?

PromQL is Prometheus's query language. It selects series by name and labels, computes rates over time windows and aggregates across labels, for example summing request rates per service.

### Can Prometheus monitor GPUs?

Yes, through exporters. The NVIDIA DCGM exporter exposes GPU utilisation, memory, temperature and power as Prometheus metrics.

### Is Prometheus enough for observability?

It covers metrics and alerting. Logs and traces need other tools, such as Loki and Tempo, often tied together through Grafana and OpenTelemetry.

### What changed in Prometheus 3.0?

It added a new interface, Remote Write 2.0, UTF-8 metric names, native OTLP ingestion and improved performance, with some breaking changes documented in a migration guide.

## Glossary

- Target: an endpoint Prometheus scrapes.
- Job: a group of targets with the same purpose.
- Scrape interval: how often targets are fetched.
- TSDB: the time-series database inside Prometheus.
- Alertmanager: the service that routes and silences alerts.
- Pushgateway: a buffer for metrics from short-lived jobs.
- Exporter: an adapter exposing third-party metrics.
- PromQL: the Prometheus query language.
- Native histogram: a histogram with automatic exponential buckets.
- OTLP: the OpenTelemetry Protocol for sending telemetry.

## Summary

Prometheus collects metrics by scraping HTTP endpoints, stores them as labelled time series, answers questions with PromQL and raises alerts through Alertmanager. Started at SoundCloud in 2012, it joined the CNCF in 2016, graduated in 2018 and reached version 3.0 in November 2024 with OTLP ingestion and a new interface. It is the standard metrics layer for Kubernetes and model-serving systems. Keep labels bounded, use histograms for latency, alert on user-facing symptoms and add long-term storage when you need history.