Grafana Loki

You are a DevOps engineer setting up centralized log aggregation. The user wants to deploy Grafana Loki, configure log sources, and query logs via LogQL.

What to check first

• Verify Docker and Docker Compose are installed: docker --version && docker-compose --version
• Check available disk space for log storage: df -h /var/lib/docker (minimum 10GB recommended)
• Ensure ports 3000 (Grafana) and 3100 (Loki) are not in use: netstat -tuln | grep -E ':(3000|3100)'

Steps

1. Create a docker-compose.yml with Loki, Promtail (agent), and Grafana services
2. Configure Loki's loki-config.yaml with storage backend (filesystem or S3), retention policy, and schema config
3. Set up Promtail's promtail-config.yaml to scrape logs from /var/log and other sources using job configs
4. Start services with docker-compose up -d and verify health: curl http://localhost:3100/ready
5. Add Loki as a data source in Grafana (http://localhost:3000) with URL http://loki:3100
6. Create a dashboard or use Explore tab to run LogQL queries like {job="varlogs"} or {job="docker"} | json | level="error"
7. Test label discovery with /loki/api/v1/labels endpoint to confirm Promtail is shipping logs
8. Configure retention and storage limits in loki-config.yaml limits_config section to prevent disk overflow

Code

# docker-compose.yml
version: '3.8'

services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yaml:/etc/loki/local-config.yaml
      - loki-storage:/loki
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - loki-network

  promtail:
    image: grafana/promtail:latest
    volumes:
      - ./promtail-config.yaml:/etc/promtail/config.yaml
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock
    command: -config.file=/etc/promtail/config.yaml
    depends_on:
      - loki
    networks:
      - loki-network

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD

Note: this example was truncated in the source. See the GitHub repo for the latest full version.

Common Pitfalls

• Treating this skill as a one-shot solution — most workflows need iteration and verification
• Skipping the verification steps — you don't know it worked until you measure
• Applying this skill without understanding the underlying problem — read the related docs first

When NOT to Use This Skill

• When a simpler manual approach would take less than 10 minutes
• On critical production systems without testing in staging first
• When you don't have permission or authorization to make these changes

How to Verify It Worked

• Run the verification steps documented above
• Compare the output against your expected baseline
• Check logs for any warnings or errors — silent failures are the worst kind

Production Considerations

• Test in staging before deploying to production
• Have a rollback plan — every change should be reversible
• Monitor the affected systems for at least 24 hours after the change