Centralized Log Management with Grafana Loki on a VPS

20 min read·Matthieu·MonitoringLogQLLoggingDocker ComposePromtailLokiGrafana|

Deploy Grafana Loki, Promtail, and Grafana via Docker Compose on a single VPS. Collect systemd, Docker, and Nginx logs, query them with LogQL, and configure retention for production use.

Scattered logs across /var/log and docker logs output stop being manageable once you run more than two services. This tutorial deploys the Grafana + Loki + Promtail stack on a single VPS using Docker Compose. You will collect logs from systemd, Docker containers, and Nginx, query them with LogQL, configure retention so logs do not fill your disk, and lock the stack down for a public-facing server.

By the end, Loki's HTTP API will be ready for programmatic queries. The AI Log Analysis with Ollama on a VPS: Detect Anomalies with a Local LLM article builds on this to feed logs into a local LLM for anomaly detection.

What does the Grafana + Loki + Promtail stack do?

Grafana Loki is an open-source log aggregation system that indexes only log labels (metadata), not the full text of log entries. This makes it far lighter on resources than Elasticsearch. Loki stores compressed log chunks on the filesystem or object storage. Paired with Promtail for collection and Grafana for visualization, it forms a complete centralized logging stack.

The three components have distinct roles:

Component Role Resource footprint
Loki Receives, stores, and indexes logs. Serves queries. 300-600 MB RAM idle, up to 1 GB under heavy queries
Promtail Discovers log sources, tails files, ships entries to Loki 50-100 MB RAM
Grafana Web UI for querying and visualizing logs via Explore 200-300 MB RAM

Total stack footprint: 1-1.5 GB RAM. A 2 GB VPS is the minimum. A 4 GB VPS gives comfortable headroom for LogQL queries on larger datasets.

Loki vs Elasticsearch: Elasticsearch indexes every word in every log line, which gives full-text search but costs 10-20x more RAM and disk. Loki's label-only index means you filter by labels first, then grep through the matching chunks. For most VPS workloads, this is the right trade-off. If you need full-text search across terabytes of logs, Loki is not the right tool.

Prerequisites

  • A VPS with at least 2 GB RAM (4 GB recommended). A Virtua Cloud VPS with 4 vCPU and 8 GB RAM handles this stack with room to spare.
  • Docker and Docker Compose installed. If you need setup help, see Docker Compose for Multi-Service VPS Deployments.
  • A non-root user with sudo access.
  • Basic familiarity with the terminal and YAML syntax.

Verify Docker is running:

docker --version
docker compose version

You should see Docker 24+ and Compose v2+. If either command fails, Docker is not installed or the Compose plugin is missing.

How do I deploy Loki with Docker Compose on a VPS?

Create a project directory and three configuration files: docker-compose.yml, loki-config.yml, and promtail-config.yml. The Docker Compose file pins all images to specific versions, sets resource limits, configures persistent volumes, and binds Loki to localhost only.

Project structure

mkdir -p ~/loki-stack/{loki-data,promtail-data}
cd ~/loki-stack

The loki-data directory holds chunks, indexes, and the write-ahead log. The promtail-data directory stores Promtail's position file so it can resume after restarts.

Set ownership on the Loki data directory. The Loki 3.x Docker image runs as UID 10001, not root. Without this, Loki fails to start with "permission denied" when creating subdirectories:

chown 10001:10001 ~/loki-stack/loki-data

docker-compose.yml

services:
  loki:
    image: grafana/loki:3.6.7
    command: -config.file=/etc/loki/config.yml
    volumes:
      - ./loki-config.yml:/etc/loki/config.yml:ro
      - ./loki-data:/loki
    ports:
      - "127.0.0.1:3100:3100"
    restart: unless-stopped
    deploy:
      resources:
        limits:
          memory: 1g
    networks:
      - loki-net

  promtail:
    image: grafana/promtail:3.6.7
    command: -config.file=/etc/promtail/config.yml
    volumes:
      - ./promtail-config.yml:/etc/promtail/config.yml:ro
      - ./promtail-data:/var/lib/promtail
      - /var/log:/var/log:ro
      - /var/log/journal:/var/log/journal:ro
      - /run/log/journal:/run/log/journal:ro
      - /etc/machine-id:/etc/machine-id:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    restart: unless-stopped
    deploy:
      resources:
        limits:
          memory: 256m
    depends_on:
      - loki
    networks:
      - loki-net

  grafana:
    image: grafana/grafana:11.5.2
    volumes:
      - grafana-data:/var/lib/grafana
    ports:
      - "127.0.0.1:3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/grafana_admin_pw
      - GF_SERVER_ROOT_URL=http://localhost:3000
      - GF_USERS_ALLOW_SIGN_UP=false
      - GF_ANALYTICS_REPORTING_ENABLED=false
    secrets:
      - grafana_admin_pw
    restart: unless-stopped
    deploy:
      resources:
        limits:
          memory: 512m
    depends_on:
      - loki
    networks:
      - loki-net

secrets:
  grafana_admin_pw:
    file: ./secrets/grafana_admin_pw

volumes:
  grafana-data:

networks:
  loki-net:
    driver: bridge

Sharp eyes: notice Loki and Grafana bind to 127.0.0.1, not 0.0.0.0. This prevents external access. You will access Grafana through an SSH tunnel or a reverse proxy. Exposing Loki or Grafana directly to the internet is a common mistake that appears in most tutorials online.

The restart: unless-stopped policy ensures every service survives reboots. If you manually docker compose stop a service, it stays stopped. Otherwise it restarts automatically.

Pinned image tags (3.6.7, 11.5.2) prevent surprise upgrades. Never use :latest in production. When you want to upgrade, change the tag and run docker compose up -d to pull the new image.

Generate the Grafana admin password

Never use default credentials. Generate a strong password and store it in a secrets file with restricted permissions:

mkdir -p ~/loki-stack/secrets
openssl rand -base64 32 > ~/loki-stack/secrets/grafana_admin_pw
chmod 644 ~/loki-stack/secrets/grafana_admin_pw

Verify the permissions:

ls -la ~/loki-stack/secrets/grafana_admin_pw

You should see -rw-r--r--. The file needs to be world-readable because Docker Compose (outside Swarm mode) bind-mounts file-based secrets with the source file's permissions. Grafana runs as UID 472 inside the container, so it needs read access. The file is still protected by its location in a dedicated secrets directory, and only the host's root user can modify it. The GF_SECURITY_ADMIN_PASSWORD__FILE environment variable tells Grafana to read the password from this file at startup rather than embedding it in the compose file.

loki-config.yml

This configuration uses TSDB with schema v13 (Loki 3.x defaults), filesystem storage for single-node deployment, and the compactor for retention:

auth_enabled: false

server:
  http_listen_port: 3100
  http_listen_address: 0.0.0.0
  log_level: warn
  http_server_read_timeout: 30s
  http_server_write_timeout: 30s

common:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory
  replication_factor: 1
  path_prefix: /loki

schema_config:
  configs:
    - from: "2024-01-01"
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

storage_config:
  filesystem:
    directory: /loki/chunks
  tsdb_shipper:
    active_index_directory: /loki/tsdb-index
    cache_location: /loki/tsdb-cache

compactor:
  working_directory: /loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  delete_request_store: filesystem

limits_config:
  retention_period: 720h
  max_streams_per_user: 10000
  ingestion_rate_mb: 16
  ingestion_burst_size_mb: 32
  max_label_names_per_series: 15

ingester:
  chunk_encoding: snappy
  wal:
    dir: /loki/wal
    enabled: true

chunk_store_config:
  chunk_cache_config:
    embedded_cache:
      enabled: true
      max_size_mb: 100

Key choices in this config:

  • auth_enabled: false is safe here because Loki only listens on localhost (Docker internal network + 127.0.0.1 port binding). Multi-tenant setups need auth_enabled: true with an X-Scope-OrgID header on every request.
  • retention_period: 720h keeps logs for 30 days. Loki 3.x defaults to 0s (keep forever) if you do not set this. Your disk will fill up.
  • schema: v13 with store: tsdb is required for Loki 3.x features. Older boltdb-shipper configs from Loki 2.x tutorials will fail to start or produce deprecation warnings.
  • chunk_encoding: snappy compresses chunks with Snappy. Faster than gzip, slightly larger files. Good default for single-node where CPU is more constrained than disk.
  • WAL enabled: the write-ahead log protects against data loss if Loki crashes mid-write. On restart, Loki replays the WAL to recover uncommitted entries. You will see "WAL replay" messages in the logs at startup. This is normal.
  • max_label_names_per_series: 15 matches the Loki 3.x default. Keep label cardinality low. Labels like user_id or request_id create too many streams and degrade performance.

How do I configure Promtail to collect systemd journal logs?

Promtail scrapes logs from multiple sources and ships them to Loki. The configuration below collects from three sources in separate scrape_configs jobs: systemd journal, Docker containers, and Nginx log files.

Promtail EOL notice: Promtail reached end-of-life on March 2, 2026. Grafana Alloy is the official successor. This tutorial uses Promtail because the configuration concepts transfer directly to Alloy, and millions of deployments still run it. See the "Is Promtail deprecated?" section below for migration steps.

promtail-config.yml

server:
  http_listen_port: 9080
  log_level: warn

positions:
  filename: /var/lib/promtail/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  # --- Systemd journal ---
  - job_name: journal
    journal:
      max_age: 12h
      labels:
        job: systemd-journal
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: unit
      - source_labels: ['__journal_priority_keyword']
        target_label: severity
      - source_labels: ['__journal__hostname']
        target_label: hostname

  # --- Docker containers ---
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.+)'
        target_label: container
      - source_labels: ['__meta_docker_container_log_stream']
        target_label: stream
      - source_labels: ['__meta_docker_container_label_com_docker_compose_service']
        target_label: compose_service

  # --- Nginx access and error logs ---
  - job_name: nginx
    static_configs:
      - targets:
          - localhost
        labels:
          job: nginx
          type: access
          __path__: /var/log/nginx/access.log
      - targets:
          - localhost
        labels:
          job: nginx
          type: error
          __path__: /var/log/nginx/error.log

The positions.yaml file tracks how far Promtail has read in each log source. If Promtail restarts, it picks up where it left off instead of re-sending old logs or missing new ones.

How the journal job works

The journal block reads directly from the systemd journal via the journal API. The max_age: 12h setting tells Promtail to only ingest journal entries from the last 12 hours on first startup. Without this, Promtail would try to ingest the entire journal history, which can be gigabytes on long-running servers.

The relabel_configs extract metadata from journal entries into Loki labels. __journal__systemd_unit becomes the unit label (e.g., sshd.service, nginx.service). __journal_priority_keyword becomes severity (e.g., warning, err, info). These labels let you filter logs efficiently in LogQL without scanning every line.

For journal scraping to work, the Promtail container needs two volume mounts: /run/log/journal (or /var/log/journal if your system persists logs) and /etc/machine-id. The machine ID identifies which journal to read.

Docker image limitation: The standard grafana/promtail Docker image is not compiled with systemd journal support. If you see support for reading the systemd journal is not compiled into this build of promtail in the logs, journal scraping will not work from the Docker container. You have two options:

  1. Install Promtail as a host binary instead of using the Docker image. Download it from the Loki releases page and run it directly on the host, where it has native access to the journal API.
  2. Use Grafana Alloy (see the migration section below), which supports journal scraping in its Docker image.

The Docker and Nginx file scraping jobs work fine in the Docker image. Only journal scraping requires the host binary or Alloy.

How do I collect Docker container logs with Promtail?

The docker job uses Docker service discovery to auto-discover all running containers on the host. Promtail connects to the Docker daemon via /var/run/docker.sock and polls for new containers every 5 seconds. When a container starts or stops, Promtail automatically starts or stops tailing its logs.

The relabel_configs extract useful metadata:

  • __meta_docker_container_name becomes the container label. The regex '/(.+)' strips the leading / that Docker adds to container names.
  • __meta_docker_container_log_stream becomes the stream label (stdout or stderr).
  • __meta_docker_container_label_com_docker_compose_service extracts the Compose service name (e.g., loki, grafana). This label only exists for containers managed by Docker Compose.

You do not need to configure individual containers. Every container with a log driver that writes to the filesystem (the default json-file driver) will be discovered. If you run a database container, a web app, and a cache, all three appear automatically in Loki under their container names.

To exclude specific containers from log collection, add a Docker label and filter on it:

    relabel_configs:
      # ... existing relabel rules ...
      - source_labels: ['__meta_docker_container_label_logging']
        regex: 'disabled'
        action: drop

Then on the container you want to exclude:

  noisy-service:
    image: some/image
    labels:
      logging: "disabled"

How do I scrape Nginx access and error logs with Promtail?

The nginx job uses static_configs with the __path__ label to tail specific log files. Unlike Docker service discovery, this requires you to know the log file paths in advance. Nginx writes to /var/log/nginx/access.log and /var/log/nginx/error.log by default.

The type label distinguishes between access and error logs. This lets you query them separately in LogQL:

{job="nginx", type="access"}    # only access logs
{job="nginx", type="error"}     # only error logs
{job="nginx"}                   # both

If Nginx is not installed on the host, Promtail logs a warning about missing files but continues scraping other sources. This is harmless. Remove the nginx job from the config if you do not use Nginx.

For Nginx running inside a Docker container, you have two options. You can use the Docker service discovery (the container's stdout/stderr will be captured automatically). Or you can mount the Nginx log directory as a shared volume and use the static file scraper. The Docker approach is simpler. The file approach gives you separate type: access and type: error labels.

Start the stack

cd ~/loki-stack
docker compose up -d

Verify all three containers are running:

docker compose ps

You should see loki, promtail, and grafana with status Up. If any service shows Restarting, check its logs:

docker compose logs <service-name> --tail=30

Check Loki is ready:

curl -s http://127.0.0.1:3100/ready

Expected output: ready. If you get Ingester not ready: waiting for 15s after being ready, wait 15 seconds and retry. Loki needs time to initialize the ingester ring.

Check Promtail targets:

docker compose logs promtail --tail=20

Look for lines showing discovered targets. You should see entries for the journal, Docker socket, and Nginx log paths. No level=error lines should appear.

How do I verify logs appear in Grafana Explore?

Open an SSH tunnel to access Grafana from your local machine. We use a tunnel because Grafana binds to localhost on the VPS and is not exposed to the internet.

ssh -L 3000:127.0.0.1:3000 user@your-vps-ip

Open http://localhost:3000 in your browser. Log in with username admin and the password from ~/loki-stack/secrets/grafana_admin_pw. Read it with:

cat ~/loki-stack/secrets/grafana_admin_pw

Add Loki as a data source

  1. Go to Connections > Data Sources > Add data source
  2. Select Loki
  3. Set the URL to http://loki:3100 (this is the Docker internal hostname, not localhost)
  4. Click Save & test

You should see "Data source successfully connected." If it fails, verify that both containers are on the same Docker network (loki-net).

Run your first query

  1. Go to Explore (compass icon in the sidebar)
  2. Select the Loki data source
  3. Switch to Code mode (not Builder) and enter this LogQL query:
{job="systemd-journal"} |= "ssh"

This shows all systemd journal entries containing "ssh". If you see log lines, the full pipeline is working: journal -> Promtail -> Loki -> Grafana.

Try a Docker container query:

{compose_service="loki"}

This returns Loki's own logs, collected by Promtail through Docker service discovery.

And an Nginx query (if Nginx is installed and generating logs):

{job="nginx", type="error"}

If Grafana shows "No data," wait 2-3 minutes. Loki needs time to ingest and index the first batch of logs.

What are the most useful LogQL queries for server logs?

LogQL has two query types: log queries return log lines, metric queries return numeric values. Both start with a stream selector ({label="value"}) that picks which log streams to scan, then add filters and parsers to refine results.

Stream selectors and line filters

# All logs from the SSH daemon
{unit="sshd.service"}

# Lines containing "Failed password"
{unit="sshd.service"} |= "Failed password"

# Lines NOT containing "Accepted"
{unit="sshd.service"} != "Accepted"

# Regex match: IP addresses
{unit="sshd.service"} |~ "\\d+\\.\\d+\\.\\d+\\.\\d+"

# Case-insensitive match
{job="nginx", type="error"} |~ "(?i)timeout"

Line filters (|=, !=, |~, !~) run after the stream selector. They scan log line content. Multiple filters chain together and all must match:

{unit="sshd.service"} |= "Failed" |= "root"

This finds lines containing both "Failed" and "root".

Parsers

Parsers extract structured fields from unstructured log lines. Once parsed, you can filter on extracted fields like status >= 500 instead of regex matching. Choose the right parser for your log format:

Parser Syntax Best for Performance notes
logfmt | logfmt Key=value logs (systemd, Go apps) Fastest. Zero regex.
json | json JSON-structured logs Fast. Native JSON parsing.
pattern | pattern "<pattern>" Fixed-format logs (Nginx combined, Apache) Fast. Positional extraction.
regexp | regexp "<regex>" Irregular formats, mixed structures Slowest. Use as last resort.

Use logfmt or json when your logs already have structure. Use pattern for well-known formats like Nginx combined log. Use regexp only when nothing else works, because regex parsing is significantly slower on high-volume streams.

Nginx access log with pattern parser:

{job="nginx", type="access"}
  | pattern "<ip> - - [<timestamp>] \"<method> <uri> <_>\" <status> <bytes>"
  | status >= 500

This parses the Nginx combined log format into named fields and filters for 5xx errors. The <_> placeholder discards fields you do not need (the HTTP version, in this case).

JSON log parser:

{compose_service="myapp"}
  | json
  | level="error"
  | line_format "{{.timestamp}} {{.message}}"

The line_format stage reformats the output. Useful when JSON logs are noisy and you want cleaner output in Grafana.

Metrics from logs

Metric queries turn log lines into numbers. These power Grafana dashboards and alerting rules:

# Failed SSH login rate per minute over the last hour
rate({unit="sshd.service"} |= "Failed password" [5m])

# Total Nginx 5xx errors in 5-minute windows
count_over_time(
  {job="nginx", type="access"}
    | pattern "<_> - - [<_>] \"<_> <_> <_>\" <status> <_>"
    | status >= 500
  [5m]
)

# P95 response time from JSON app logs
# unwrap extracts a numeric field for aggregation
quantile_over_time(0.95,
  {compose_service="myapp"}
    | json
    | unwrap response_time_ms
    | __error__=""
  [5m]
) by (endpoint)

# Bytes served per second by Nginx
sum(rate(
  {job="nginx", type="access"}
    | pattern "<_> - - [<_>] \"<_> <_> <_>\" <_> <bytes>"
    | unwrap bytes
    | __error__=""
  [5m]
))

The | __error__="" filter after unwrap drops lines where numeric extraction failed (non-numeric values, missing fields). Without it, those lines silently produce zero values and skew your results. Always include this filter after unwrap.

The [5m] range defines the window size. Shorter ranges (1m) give more granular data but are noisier. Longer ranges (15m, 1h) smooth out spikes. For dashboards, 5m is a good starting point.

How do I set log retention in Loki 3.x?

In Loki 3.x, retention is managed by the compactor. Set retention_period under limits_config and enable the compactor with retention_enabled: true. The default retention in Loki 3.0+ is 0s (keep forever), so you must configure this explicitly or your disk fills up.

The loki-config.yml above already includes retention. Here is how the settings interact:

compactor:
  retention_enabled: true        # Must be true, or compactor only compacts (no deletion)
  retention_delete_delay: 2h     # Wait 2h after marking chunks before deleting
  compaction_interval: 10m       # How often the compactor runs

limits_config:
  retention_period: 720h         # 30 days global default

The compactor runs as part of the Loki process in single-node mode. It scans the TSDB index, identifies chunks older than the retention period, marks them for deletion, then removes them after the retention_delete_delay. The delay gives you a window to recover if retention is misconfigured.

Per-stream retention

You can set different retention periods for different log streams. High-volume, low-value logs (like debug output) can expire faster:

limits_config:
  retention_period: 720h
  retention_stream:
    - selector: '{job="nginx", type="access"}'
      priority: 1
      period: 336h    # 14 days for access logs
    - selector: '{severity="debug"}'
      priority: 2
      period: 72h     # 3 days for debug logs

Higher priority values win when multiple selectors match the same stream. A debug-level Nginx access log matches both rules. Priority 2 wins, so it gets 3-day retention.

Retention sizing

Estimate disk usage before committing to a retention period. Loki compresses logs well (5-10x ratio with Snappy), but the numbers add up on busy servers:

Raw log volume/day Compressed (est.) 7-day retention 30-day retention 90-day retention
100 MB ~15 MB ~105 MB ~450 MB ~1.35 GB
500 MB ~75 MB ~525 MB ~2.25 GB ~6.75 GB
1 GB ~150 MB ~1.05 GB ~4.5 GB ~13.5 GB
5 GB ~750 MB ~5.25 GB ~22.5 GB ~67.5 GB

These are estimates. Actual compression depends on log content. Repetitive logs (access logs with similar paths) compress better than random debug output. Monitor real usage with:

du -sh ~/loki-stack/loki-data/

Run this weekly to catch unexpected growth before you run out of disk space.

How do I tune Loki for production on a single VPS?

A default Loki config works for testing but needs tuning for a production VPS. The changes below reduce memory spikes, protect against runaway log streams, and harden the stack for a public-facing server.

Bind Loki to localhost

Already done in the docker-compose.yml above (127.0.0.1:3100:3100). Double-check after deployment:

ss -tlnp | grep 3100

You should see 127.0.0.1:3100, not 0.0.0.0:3100. Do the same for Grafana on port 3000.

Firewall rules

If you use ufw, block external access to the logging ports:

sudo ufw deny 3100/tcp comment "Loki - localhost only"
sudo ufw deny 3000/tcp comment "Grafana - localhost only"
sudo ufw status numbered

Since the ports already bind to localhost, the firewall is defense in depth. If someone accidentally changes the compose file to bind to 0.0.0.0, the firewall still blocks external access.

Version information hiding

Version disclosure helps attackers target known vulnerabilities. The GF_ANALYTICS_REPORTING_ENABLED=false in the compose file already disables Grafana telemetry. Loki's /loki/api/v1/status/buildinfo endpoint exposes version details, but since Loki is bound to localhost, only local processes can reach it.

If you put Grafana behind a reverse proxy (Nginx, Caddy), add these Nginx settings:

server_tokens off;
proxy_hide_header X-Powered-By;

Key Loki 3.x breaking changes

If you are migrating from a Loki 2.x config or following an older tutorial, be aware of these changes:

Change Loki 2.x default Loki 3.x default Action required
Schema v11/v12 v13 Use schema: v13 with store: tsdb
Index store boltdb-shipper tsdb Migrate to TSDB (BoltDB deprecated)
Retention 0s (keep forever) 0s (keep forever) Set retention_period explicitly
Structured metadata Disabled Enabled Requires v13 schema
Max labels per series 30 15 Reduce label cardinality or increase limit
Docker image Includes BusyBox shell No shell Cannot docker exec into container

The no-shell Docker image in Loki 3.6+ means you cannot run docker exec -it loki sh for debugging. Instead, check logs with docker compose logs loki and readiness with curl http://127.0.0.1:3100/ready.

Filesystem vs object storage

For a single VPS, filesystem storage is correct. Object storage (S3, GCS, MinIO) adds complexity and latency that only pays off when you need:

  • Multiple Loki instances sharing the same data
  • Unlimited storage beyond your VPS disk
  • Cross-region replication

Stick with filesystem storage until you outgrow a single node.

Monitor the stack

Check resource usage of the running containers:

docker stats --no-stream

Check Loki logs for warnings and errors:

docker compose logs loki --tail=50 | grep -E "level=(error|warn)"

Common issues and fixes:

  • stream limit exceeded: increase max_streams_per_user in limits_config. Usually caused by high-cardinality labels.
  • ingestion rate limit reached: increase ingestion_rate_mb. Happens during log bursts (deployments, error storms).
  • WAL replay: normal at startup. Loki is recovering uncommitted writes from the write-ahead log.
  • High memory usage: reduce max_size_mb in chunk_cache_config or lower ingestion_burst_size_mb.

Is Promtail deprecated? Should I use Grafana Alloy instead?

Yes. Promtail entered Long-Term Support on February 13, 2025 and reached end-of-life on March 2, 2026. No future updates, bug fixes, or security patches will be released. Grafana Alloy is the official replacement. It is Grafana Labs' distribution of the OpenTelemetry Collector and handles logs, metrics, traces, and profiling data in a single agent.

Why this tutorial still uses Promtail

Promtail configuration concepts map directly to Alloy. The scrape configs, relabel rules, and pipeline stages work the same way. Learning Promtail is still useful because:

  1. Millions of existing deployments run it
  2. The Alloy migration tool converts Promtail configs automatically
  3. Understanding Promtail makes debugging Alloy configurations easier
  4. Search volume for Promtail tutorials remains high, and the concepts transfer

Migrating to Alloy

Convert your Promtail config to Alloy format with one command:

alloy convert --source-format=promtail --output=alloy-config.alloy promtail-config.yml

This generates an Alloy-format config file. Review the output before deploying. The converter handles most cases but may need manual adjustments for custom pipeline stages.

Then replace the Promtail service in docker-compose.yml:

  alloy:
    image: grafana/alloy:v1.14.1
    command:
      - run
      - /etc/alloy/config.alloy
      - --server.http.listen-addr=0.0.0.0:12345
    volumes:
      - ./alloy-config.alloy:/etc/alloy/config.alloy:ro
      - /var/log:/var/log:ro
      - /var/log/journal:/var/log/journal:ro
      - /run/log/journal:/run/log/journal:ro
      - /etc/machine-id:/etc/machine-id:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    restart: unless-stopped
    depends_on:
      - loki
    networks:
      - loki-net

Remove the promtail service block and run docker compose up -d. Alloy will start collecting the same log sources using the converted config.

For new deployments starting from scratch, use Alloy from the beginning. For existing Promtail setups, plan the migration but there is no immediate urgency. The Promtail binary continues to work. Pin the image tag (grafana/promtail:3.6.7) so you control what runs.

How do I query Loki logs programmatically via the HTTP API?

Loki exposes a REST API for programmatic log queries. This is how you integrate Loki with scripts, alerting pipelines, or the AI log analysis layer covered in AI Log Analysis with Ollama on a VPS: Detect Anomalies with a Local LLM.

The API accepts the same LogQL queries you use in Grafana. The main endpoint is /loki/api/v1/query_range for time-range queries.

Query recent logs

END=$(date +%s)000000000
START=$(( $(date +%s) - 3600 ))000000000

curl -s "http://127.0.0.1:3100/loki/api/v1/query_range" \
  --data-urlencode "query={unit=\"sshd.service\"} |= \"Failed password\"" \
  --data-urlencode "start=$START" \
  --data-urlencode "end=$END" \
  --data-urlencode "limit=50" | jq .

The start and end parameters use Unix nanosecond timestamps. The shell arithmetic above calculates "now minus 1 hour" and appends nine zeros for nanosecond precision.

The response is JSON:

{
  "status": "success",
  "data": {
    "resultType": "streams",
    "result": [
      {
        "stream": {"unit": "sshd.service", "severity": "info"},
        "values": [
          ["1710850000000000000", "Mar 19 14:00:00 vps sshd[1234]: Failed password for root from 203.0.113.5 port 22"]
        ]
      }
    ]
  }
}

Each value is a [timestamp_nanoseconds, log_line] pair. This is the exact format you will parse when feeding logs into a local LLM for analysis.

Query labels and streams

List all label names:

curl -s "http://127.0.0.1:3100/loki/api/v1/labels" | jq .

List values for a specific label:

curl -s "http://127.0.0.1:3100/loki/api/v1/label/unit/values" | jq .

These endpoints are useful for building dynamic queries in automation scripts. You can enumerate all units, then query each one for errors.

Instant queries

For "right now" queries without a time range, use /loki/api/v1/query:

curl -s "http://127.0.0.1:3100/loki/api/v1/query" \
  --data-urlencode 'query=count_over_time({job="systemd-journal"} |= "error" [1h])' | jq .

This returns a single data point: the count of journal errors in the last hour. Useful for health checks and monitoring scripts.

Troubleshooting

Promtail shows "permission denied" for Docker socket:

The Promtail container needs read access to /var/run/docker.sock. Check the socket permissions on the host:

ls -la /var/run/docker.sock

The socket is typically owned by root:docker. The Promtail image runs as root by default, so this usually works. If you run Promtail with a custom user, that user must be in the docker group.

No journal logs appearing:

First, check Promtail logs for the message support for reading the systemd journal is not compiled into this build of promtail. If you see this, the Docker image does not support journal scraping. Install Promtail as a host binary or switch to Grafana Alloy (see sections above).

If Promtail is running as a host binary with journal support, verify the journal directory exists:

ls -la /var/log/journal/

If it does not exist, systemd is using volatile (memory-only) journal storage. Enable persistent storage:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

Then update the Promtail volume mount in docker-compose.yml from /run/log/journal to /var/log/journal and restart:

docker compose up -d promtail

Loki reports "too many outstanding requests":

Query load exceeds Loki's capacity. Reduce the time range of your queries or add query limits:

limits_config:
  max_query_parallelism: 16
  max_query_series: 500

Grafana shows "Data source connected, no labels found":

Loki needs a few minutes to ingest and index the first logs. Wait 2-3 minutes, then retry the query. Verify Loki is ready:

curl -s http://127.0.0.1:3100/ready

Check all service logs at once:

docker compose logs -f --tail=50

This tails all three services. Filter for problems:

docker compose logs --tail=100 | grep -i error

Next steps

Your Loki pipeline is collecting logs from systemd, Docker, and Nginx. You can query them with LogQL in Grafana or via the HTTP API.

From here:

Copyright 2026 Virtua.Cloud. All rights reserved. This content is original work by the Virtua.Cloud team. Reproduction, republication, or redistribution without written permission is prohibited.

Ready to try it yourself?

Deploy your own server in seconds. Linux, Windows, or FreeBSD. →