Time-Series Databases

Choose and optimize time-series databases for metrics, IoT, and financial data. Covers data models, compression, retention policies, downsampling, query optimization, and the patterns that handle billions of data points efficiently.

Time-series databases are optimized for storing and querying high-velocity, time-ordered data—such as metrics, logs, and sensor readings—where writes dominate and queries focus on ranges, aggregations, and downsampling over time. They matter when you need to analyze data at scale across a continuous time axis, especially when ingestion rates exceed thousands of points per second per metric, and when queries must return aggregated values across hours, days, or weeks with low latency.

Ingesting Time-Series Data

Ingestion is the primary workload in time-series databases. Data arrives in bursts, often from distributed sources like application metrics, IoT devices, or monitoring agents. The ingestion pipeline must handle high write throughput, variable data rates, and event-ordering across sources.

Each data point consists of a timestamp, a metric name (or series key), and one or more values. The most common ingestion format is a line-oriented protocol: one data point per line, with timestamp, series, and value(s) in a structured layout.

cpu_usage,host=server01,region=us-west value=78.4 1620000000
memory_used,host=server01,region=us-west value=1048576 1620000000
disk_io,host=server02,region=us-east value=12345 1620000000

This format—commonly known as the Line Protocol—is used by InfluxDB, TimescaleDB, OpenTSDB, and others. The timestamp is in Unix epoch seconds (or milliseconds), and tags (key-value pairs) form the series identifier.

A typical ingestion workflow uses a push-based API. A client sends data via HTTP POST with a body containing the Line Protocol.

curl -X POST \
  -H "Content-Type: text/plain; charset=utf-8" \
  -d 'cpu_usage,host=server01,region=us-west value=78.4 1620000000
memory_used,host=server01,region=us-west value=1048576 1620000000' \
  http://tsdb.example.com/write?db=metrics

The db query parameter selects the database (or schema), and the server responds with HTTP status 204 (No Content) on success.

Key Ingestion Challenges

Silent Failures

{
  "error": "partial write",
  "errorLines": [
    {
      "line": 123,
      "timestamp": 1620000123,
      "error": "invalid series key"
    },
    {
      "line": 456,
      "timestamp": 1620000456,
      "error": "missing required field: value"
    }
  ]
}

Querying Time-Series Data

Queries are the primary way to analyze data. They focus on time ranges, aggregations, and windowed operations.

The most common query interface is a SQL-like language with time-based clauses and built-in functions. Queries return a table where each row is a time bucket, and columns represent series or aggregations.

SELECT
  mean(value) AS avg_cpu,
  median(value) AS med_mem
FROM cpu_usage
WHERE
  time >= '2021-05-01T00:00:00Z'
  AND time < '2021-05-02T00:00:00Z'
  AND host = 'server01'
GROUP BY
  time(5m)

This query computes the 5-minute mean of CPU usage for server01, returning one row per 5-minute window.

Time-Windowing and Aggregations

Time-windowing uses GROUP BY time(interval) to bucket data. The interval can be in seconds, minutes, hours, or days.

Aggregation functions like mean, sum, count, min, max, last, and first are evaluated within each window.

Common Query Patterns

SELECT
  moving_average(value, 10m) AS rolling_avg
FROM cpu_usage
WHERE time >= now() - 1h
SELECT
  mean(value) AS avg,
  count(value) AS points
FROM cpu_usage
WHERE time >= '2021-05-01T00:00:00Z'
  AND time < '2021-05-02T00:00:00Z'
GROUP BY time(1h)

This query takes 1-second data and produces hourly aggregates.

SELECT
  mean(value)
FROM metrics
WHERE
  time >= '2021-05-01T00:00:00Z'
  AND region = 'us-west'
  AND host =~ 'server[0-9]+'
GROUP BY time(15m)

The =~ operator matches a regular expression on the host tag.

Failure Modes

Managing Data Retention and Storage

Time-series databases typically store data in a columnar format, with time as the primary dimension. Data is compressed and stored in chunks, often partitioned by time.

Retention policies define how long data is kept and at what resolution.

CREATE RETENTION POLICY monthly
  ON metrics
  DURATION 180d
  REPLICATION 3
  DEFAULT;

This policy applies to the metrics database and keeps data for 180 days. Data is stored in 1-second granularity by default, but can be downsampled to 1-minute and 1-hour granularities.

The database automatically moves data from high-resolution storage to lower-resolution storage over time.

Data Lifecycle

The transition is triggered by a scheduler or a background process that scans for data older than a threshold.

Configuration Keys

storage:
  chunk_size: 10000
  compression: zstd
  cache_size: 256MB
  retention:
    default:
      duration: 180d
      replication: 3
      tier: hot
    monthly:
      duration: 365d
      replication: 2
      tier: warm

Silent Failure: Retention Policy Misalignment

A common failure occurs when the retention policy applies to the wrong time interval. A policy created for 1h resolution but applied to 1s data results in data being downsampled too often. The database may use time(1h) but the data has a time column with 1-second resolution. The aggregation over 1-hour buckets then includes 3,600 points per bucket, but the downsampled data is not aligned to the original resolution.

The database may also fail to apply retention policies correctly when time zones differ between the ingestion and storage systems. If data is ingested in UTC but retention policies are applied in local time, a daily retention policy may start at 2021-05-01T00:00:00 in UTC, but the data is stored in a local time zone, leading to misaligned boundaries.

Scaling and High Availability

Time-series databases are often deployed in clusters. Data is sharded across nodes using a hash or range partitioning on the series key.

A typical cluster setup uses a coordinator node that routes queries and manages metadata.

Query Routing

When a client sends a query to the cluster, the coordinator node parses the query, identifies which nodes contain the required data, and sends subqueries to each node.

{
  "query": "SELECT mean(value) FROM cpu_usage WHERE time >= '2021-05-01T00:00:00Z' AND host = 'server01'",
  "targets": [
    "tsdb-node-01:8086",
    "tsdb-node-02:8086"
  ],
  "shards": [
    { "series": "cpu_usage", "start": "2021-05-01T00:00:00Z", "end": "2021-05-02T00:00:00Z" }
  ]
}

The coordinator collects results from each node and merges them into a single result set.

High Availability and Failover

curl -X POST \
  -H "Content-Type: text/plain" \
  -d 'cpu_usage,host=server01 value=78.4 1620000000' \
  'http://tsdb.example.com/write?db=metrics&consistency=quorum'

Failure Modes

{
  "data": [
    { "time": "2021-05-01T00:00:00Z", "avg_cpu": 78.4 },
    { "time": "2021-05-01T00:05:00Z", "avg_cpu": 79.1 }
  ],
  "meta": {
    "last_updated": "2021-05-01T00:05:30Z",
    "source": "tsdb-node-01"
  }
}

Maintenance and Monitoring

Regular maintenance tasks include compaction, index rebuilding, and data validation.

A typical maintenance script runs daily:

#!/bin/bash


# Compact data in the metrics database
curl -X POST http://tsdb.example.com/compact?db=metrics

# Rebuild series index
curl -X POST http://tsdb.example.com/rebuild-index?db=metrics

# Validate data integrity
curl -X POST http://tsdb.example.com/validate?db=metrics

The database may expose a health endpoint:

curl http://tsdb.example.com/health

Returns:

{
  "status": "ok",
  "uptime": 365200,
  "nodes": [
    { "id": "node-01", "status": "up", "last_heartbeat": 1620000000 },
    { "id": "node-02", "status": "down", "last_heartbeat": 1619998000 }
  ],
  "metrics": {
    "ingestion_rate": 12000,
    "query_latency": 45.2,
    "active_chunks": 124000
  }
}

Monitoring Signals

These signals are collected via systemd services, prometheus exporters, or kubectl probes in a Kubernetes environment.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.