Monitoring Stack — Setup Guide¶
This guide walks you through deploying the KMS observability stack step by step. For a full reference of OTLP options and pipeline internals, see → KMS Telemetry
Table of contents¶
- Prerequisites
- Local mode — KMS included
- External mode — existing KMS
- Explore Grafana
- Troubleshooting
Prerequisites¶
Before getting started, make sure the following tools are installed on your machine.
Required¶
-
Docker Engine ≥ 20.10 Compose profiles (
COMPOSE_PROFILES) require Docker Compose v2 and Docker Engine 20.10+. Verify your version: -
Docker Compose V2 (included in Docker Desktop 3.4+) The stack uses the
docker composecommand (with a space), not the legacydocker-compose.
Local mode only¶
openssl— used bygenerate-demo-cert.shto generate the demo TLS certificate. Verify it is available:On macOS it is pre-installed. On Linux:
apt install opensslordnf install openssl.
External mode only¶
- Your existing KMS must be configured to export telemetry via OTLP to the collector:
See → KMS Telemetry — OTLP configuration
# gRPC (recommended) KMS_OTLP_URL=http://<collector-host>:4317 # or HTTP KMS_OTLP_URL=http://<collector-host>:4318
Ports availability¶
The following ports must be free on the host before starting the stack:
| Port | Service |
|---|---|
3000 |
Grafana |
4317 |
OTel Collector (gRPC) |
4318 |
OTel Collector (HTTP) |
8428 |
VictoriaMetrics |
9998 |
KMS (local mode only) |
Local mode — KMS included¶
In this mode, the full stack starts together: KMS, OTel Collector, VictoriaMetrics, and Grafana. Use this mode for local development or to evaluate the stack end-to-end.
1. Configure .env¶
Open the .env file — this is the only file you need to edit.
Make sure the following values are set:
# ── KMS Mode / Docker profiles ───────────────────────────────────
COMPOSE_PROFILES=kms-local
KMS_MODE=local
# ── TLS certificate password (used by generate-demo-cert.sh) ─────
KMS_P12_PASSWORD=testpassword
# ── KMS topology metadata ────────────────────────────────────────
KMS_CLUSTER=cosmian-kms-local
# ── Metadata for metrics/traces ──────────────────────────────────
KMS_VERSION=latest
ENVIRONMENT=demo
# ── Grafana ──────────────────────────────────────────────────────
GRAFANA_ADMIN_USER=demo
GRAFANA_ADMIN_PASSWORD=demo
# ── Metrics retention (VictoriaMetrics, in months) ───────────────
METRICS_RETENTION_MONTHS=12
✏️ Screenshot placeholder —
.envfile open in editor,COMPOSE_PROFILESandKMS_MODEhighlighted.
2. Generate the demo TLS certificate¶
The KMS container requires a PKCS#12 certificate to serve its API over TLS.
The provided script generates a self-signed certificate valid for 10 years,
using the password defined in KMS_P12_PASSWORD:
Expected output:
Generating a 4096 bit RSA private key
.............................................++++
.............................................++++
.............................................++++
writing new private key to './certs/kms-demo.key'
-----
PKCS#12 demo certificate generated at: ./certs/kms.p12
Password: testpassword
⚠️ This certificate is for demo purposes only. Do not use it in production.
3. Start the stack¶
Docker will pull the required images and start all containers. On first run, this may take a minute depending on your network speed.
4. Verify the stack is healthy¶
All containers should show Up or Up (healthy):
You can also check the OTel Collector health endpoint:
Expected response: {"status":"Server available"}
External mode — existing KMS¶
Use this mode when you already have a running KMS instance and only want to attach the observability stack to it.
1. Configure .env¶
Only a few lines change compared to local mode:
-COMPOSE_PROFILES=kms-local
-KMS_MODE=local
+COMPOSE_PROFILES=
+KMS_MODE=external
-# KMS_P12_PASSWORD is not needed in external mode
-KMS_P12_PASSWORD=testpassword
KMS_CLUSTER=cosmian-kms-local
KMS_VERSION=latest
-ENVIRONMENT=demo
+ENVIRONMENT=production
GRAFANA_ADMIN_USER=demo
GRAFANA_ADMIN_PASSWORD=demo
METRICS_RETENTION_MONTHS=12
2. Point your KMS to the collector¶
On your existing KMS, enable OTLP export toward the collector host:
# via environment variable
KMS_OTLP_URL=http://<collector-host>:4317
# or via CLI argument
./cosmian_kms_server --otlp http://<collector-host>:4317 --enable-metering
If the KMS runs on the same machine as the stack, use
http://localhost:4317.
3. Start the stack (without KMS)¶
With COMPOSE_PROFILES left empty, Docker Compose will start only
OTel Collector, VictoriaMetrics, and Grafana — the KMS container is skipped.
4. Verify the stack is healthy¶
Expected output (no kms container):
Verify the collector is receiving data from your KMS:
# Check that otelcol_receiver_accepted_spans is incrementing
curl -s http://localhost:8888/metrics | grep otelcol_receiver_accepted_spans
Explore Grafana¶
Once the stack is running, open Grafana in your browser:
Login with:
| Field | Value |
|---|---|
| Username | value of GRAFANA_ADMIN_USER in .env (default: demo) |
| Password | value of GRAFANA_ADMIN_PASSWORD in .env (default: demo) |
Pre-provisioned dashboards are available under Dashboards → Browse:
After few seconds metrics will be available
Troubleshooting¶
Containers not starting¶
No data in Grafana dashboards¶
- Confirm the OTel Collector is up:
- Confirm VictoriaMetrics is receiving data:
- Check that your KMS is sending OTLP — look for incoming spans in collector logs:
KMS container fails to start (local mode)¶
Verify the certificate was generated:
If missing, re-run:Port conflict¶
If a port is already in use, identify the process:
Then either stop the conflicting process or edit the port mapping in docker-compose.yml.
For a complete reference of all OTLP options, log levels, and pipeline internals, see → KMS Telemetry