Monitoring and Metrics

This guide covers setting up a complete monitoring stack for FKApi using Prometheus for metrics collection and Grafana for visualization.

Overview

The monitoring stack provides:

Prometheus: Time-series metrics collection and storage
Grafana: Dashboard visualization and alerting
Redis Exporter: Redis-specific metrics
PostgreSQL Exporter: Database performance metrics
django-prometheus: Django and application metrics

Architecture

┌─────────────────────────────────────────────────┐
│                   Grafana                       │
│            (Visualization Layer)                │
└────────────────┬────────────────────────────────┘
                 │ Query metrics
                 ↓
┌─────────────────────────────────────────────────┐
│                Prometheus                       │
│              (Metrics Storage)                  │
└─┬───────┬───────┬────────┬─────────────────────┘
  │       │       │        │
  │       │       │        └─→ PostgreSQL Exporter
  │       │       │               (DB Metrics)
  │       │       │
  │       │       └─→ Redis Exporter
  │       │           (Cache Metrics)
  │       │
  │       └─→ Flower
  │           (Celery Metrics)
  │
  └─→ Django App (/metrics endpoint)
      (App Metrics)

Prerequisites

Before setting up monitoring, ensure you have:

FKApi running (see Setup Guide)
Docker and docker-compose (for containerized deployment)
Redis running (for cache metrics)
PostgreSQL running (for database metrics)

Installation

Docker Compose Setup

FKApi includes monitoring services in docker-compose.yml using the monitoring profile.

Start Monitoring Stack

# Start all services including monitoring
docker compose --profile monitoring up -d

# Or start monitoring services separately
docker compose up -d  # Start core services first
docker compose --profile monitoring up -d  # Add monitoring

Monitoring Services

The docker-compose configuration includes: Prometheus

prometheus:
  image: prom/prometheus:latest
  profiles:
    - monitoring
  volumes:
    - ../prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
    - prometheus_data:/prometheus
  command:
    - '--config.file=/etc/prometheus/prometheus.yml'
    - '--storage.tsdb.path=/prometheus'
  ports:
    - "${PROMETHEUS_PORT:-9090}:9090"

Redis Exporter

redis_exporter:
  image: oliver006/redis_exporter:latest
  profiles:
    - monitoring
  environment:
    - REDIS_ADDR=redis://redis:6379
  ports:
    - "${REDIS_EXPORTER_PORT:-9121}:9121"

PostgreSQL Exporter

postgres_exporter:
  image: prometheuscommunity/postgres-exporter:latest
  profiles:
    - monitoring
  environment:
    - DATA_SOURCE_NAME=postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@db:5432/${POSTGRES_DB}?sslmode=disable
  ports:
    - "${POSTGRES_EXPORTER_PORT:-9187}:9187"

Prometheus Configuration

Create prometheus/prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    monitor: 'fkapi-monitor'

scrape_configs:
  # Django application metrics
  - job_name: 'django'
    static_configs:
      - targets: ['web:8000']
    metrics_path: '/metrics'

  # Celery metrics via Flower
  - job_name: 'celery'
    static_configs:
      - targets: ['flower:5555']
    metrics_path: '/metrics'

  # Redis metrics
  - job_name: 'redis'
    static_configs:
      - targets: ['redis_exporter:9121']

  # PostgreSQL metrics
  - job_name: 'postgres'
    static_configs:
      - targets: ['postgres_exporter:9187']

  # Prometheus itself
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

If using systemd PostgreSQL instead of Docker, change the postgres target to host.docker.internal:9187 or your server IP.

Custom Metrics

FKApi includes custom metrics defined in core/metrics.py:

User Collection Metrics

from prometheus_client import Counter, Histogram

# Scrape counter
user_collection_scrapes_total = Counter(
    'fkapi_user_collection_scrapes_total',
    'Total number of user collection scrapes',
    ['userid', 'status']  # status: success, error, filtered
)

# Scrape duration
user_collection_scrape_duration_seconds = Histogram(
    'fkapi_user_collection_scrape_duration_seconds',
    'Time spent scraping user collections',
    ['userid'],
    buckets=[0.1, 0.5, 1.0, 2.0, 5.0, 10.0, 30.0, 60.0]
)

# Entries scraped
user_collection_entries_scraped = Histogram(
    'fkapi_user_collection_entries_scraped',
    'Number of entries scraped per collection',
    ['userid'],
    buckets=[0, 10, 50, 100, 200, 500, 1000, 2000]
)

Cache Metrics

from prometheus_client import Counter, Gauge

# Cache hit/miss counters
cache_hits = Counter(
    'fkapi_cache_hits_total',
    'Total number of cache hits',
    ['cache_type']
)

cache_misses = Counter(
    'fkapi_cache_misses_total',
    'Total number of cache misses',
    ['cache_type']
)

# Cache entries gauge
cache_entries = Gauge(
    'fkapi_cache_entries',
    'Number of entries in cache',
    ['cache_type']
)

Celery Task Metrics

from prometheus_client import Histogram

# Task duration
celery_task_duration_seconds = Histogram(
    'fkapi_celery_task_duration_seconds',
    'Duration of Celery tasks',
    ['task_name', 'status'],
    buckets=[1.0, 5.0, 10.0, 30.0, 60.0, 120.0, 300.0]
)

API Endpoint Metrics

from prometheus_client import Counter, Histogram

# Request counter
api_endpoint_requests = Counter(
    'fkapi_api_endpoint_requests_total',
    'Total API endpoint requests',
    ['endpoint', 'method', 'status_code']
)

# Response time
api_endpoint_duration_seconds = Histogram(
    'fkapi_api_endpoint_duration_seconds',
    'API endpoint response time',
    ['endpoint', 'method'],
    buckets=[0.01, 0.05, 0.1, 0.5, 1.0, 2.0, 5.0]
)

Using Custom Metrics

Instrument your code with metrics:

from core.metrics import (
    user_collection_scrapes_total,
    user_collection_scrape_duration_seconds,
    cache_hits,
    cache_misses,
)
import time

def scrape_user_collection(userid):
    """Scrape user collection with metrics."""
    start_time = time.time()
    
    try:
        # Perform scraping
        result = perform_scrape(userid)
        
        # Record success
        user_collection_scrapes_total.labels(
            userid=userid,
            status='success'
        ).inc()
        
        return result
        
    except Exception as e:
        # Record error
        user_collection_scrapes_total.labels(
            userid=userid,
            status='error'
        ).inc()
        raise
        
    finally:
        # Record duration
        duration = time.time() - start_time
        user_collection_scrape_duration_seconds.labels(
            userid=userid
        ).observe(duration)

def get_cached_data(cache_key, cache_type='default'):
    """Get data with cache metrics."""
    from django.core.cache import cache
    
    data = cache.get(cache_key)
    
    if data is not None:
        cache_hits.labels(cache_type=cache_type).inc()
        return data
    else:
        cache_misses.labels(cache_type=cache_type).inc()
        # Fetch and cache data
        data = fetch_data()
        cache.set(cache_key, data)
        return data

Grafana Setup

Install Grafana

Docker
Linux (systemd)
macOS

Add to docker-compose.yml:

grafana:
  image: grafana/grafana:latest
  profiles:
    - monitoring
  ports:
    - "3000:3000"
  environment:
    - GF_SECURITY_ADMIN_PASSWORD=admin
    - GF_USERS_ALLOW_SIGN_UP=false
  volumes:
    - grafana_data:/var/lib/grafana
    - ./grafana/dashboards:/etc/grafana/provisioning/dashboards
    - ./grafana/datasources:/etc/grafana/provisioning/datasources
  depends_on:
    - prometheus

# Add Grafana repository
sudo apt-get install -y software-properties-common
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

# Install Grafana
sudo apt-get update
sudo apt-get install grafana

# Start Grafana
sudo systemctl start grafana-server
sudo systemctl enable grafana-server

# Install via Homebrew
brew install grafana

# Start Grafana
brew services start grafana

Add Prometheus Data Source

Create Dashboard

FKApi includes a pre-built Grafana dashboard. Import it:

The dashboard includes:

Request Rate: Requests per second by endpoint
Response Time: P50, P95, P99 latencies
Error Rate: 4xx and 5xx error rates
Cache Hit Rate: Cache effectiveness
Database Performance: Query counts and durations
Celery Tasks: Task success/failure rates
Redis Metrics: Memory usage, operations/sec
System Resources: CPU, memory, disk usage

Monitoring Best Practices

Metric Selection

Focus on user-facing metrics (latency, errors)
Track resource utilization (CPU, memory, disk)
Monitor cache hit rates
Track task success/failure rates
Measure database query performance

Alerting

Set up alerts for high error rates
Alert on high latency (P95 > threshold)
Monitor disk space usage
Alert on cache connection failures
Track task queue backlogs

Dashboard Design

Group related metrics together
Use appropriate time ranges
Include both current and historical views
Add annotations for deployments
Use variables for filtering

Data Retention

Configure Prometheus retention period
Archive historical data if needed
Monitor Prometheus storage size
Consider using remote storage for long-term data

Troubleshooting

Metrics Endpoint Not Found

Error: 404 at /metrics Solution:

Verify django_prometheus is installed: pip list | grep django-prometheus
Check INSTALLED_APPS includes 'django_prometheus'
Verify URL configuration includes path('', include('django_prometheus.urls'))
Restart Django server

Prometheus Not Scraping

Error: No data in Prometheus UI Solution:

Check Prometheus targets: http://localhost:9090/targets
Verify all targets show as “UP”
Check firewall rules allow access
Verify service names in prometheus.yml match docker-compose
Check Prometheus logs: docker compose logs prometheus

Grafana Connection Failed

Error: Cannot connect to data source Solution:

Verify Prometheus is running: docker compose ps prometheus
Check Prometheus URL in Grafana (use service name for Docker)
Test Prometheus UI: http://localhost:9090
Check network connectivity between containers

Missing Metrics

Error: Some metrics not appearing Solution:

Verify exporters are running: docker compose ps
Check exporter logs for errors
Test exporter endpoints directly:
- Redis: http://localhost:9121/metrics
- PostgreSQL: http://localhost:9187/metrics
Verify Prometheus scrape configuration

Accessing Monitoring Tools

Once everything is running:

Prometheus UI: http://localhost:9090
- Query metrics
- View targets status
- Check service health
Grafana: http://localhost:3000
- View dashboards
- Create alerts
- Explore metrics
Flower (Celery): http://localhost:5555
- Monitor Celery tasks
- View worker status
- Track task history
Metrics Endpoint: http://localhost:8000/metrics
- Raw Prometheus metrics
- Verify instrumentation

Get Started

Core Concepts

Guides

Management Commands

Monitoring and Metrics

Overview

Architecture

Prerequisites

Installation

Docker Compose Setup

Start Monitoring Stack

Monitoring Services

Prometheus Configuration

Custom Metrics

User Collection Metrics

Cache Metrics

Celery Task Metrics

API Endpoint Metrics

Using Custom Metrics

Grafana Setup

Install Grafana

Add Prometheus Data Source

Create Dashboard

Monitoring Best Practices

Troubleshooting

Metrics Endpoint Not Found

Prometheus Not Scraping

Grafana Connection Failed

Missing Metrics

Accessing Monitoring Tools

Next Steps

Caching Strategy

Celery Setup

​Overview

​Architecture

​Prerequisites

​Installation

​Docker Compose Setup

​Start Monitoring Stack

​Monitoring Services

​Prometheus Configuration

​Custom Metrics

​User Collection Metrics

​Cache Metrics

​Celery Task Metrics

​API Endpoint Metrics

​Using Custom Metrics

​Grafana Setup

​Install Grafana

​Add Prometheus Data Source

​Create Dashboard

​Monitoring Best Practices

​Troubleshooting

​Metrics Endpoint Not Found

​Prometheus Not Scraping

​Grafana Connection Failed

​Missing Metrics

​Accessing Monitoring Tools

​Next Steps

Caching Strategy

Celery Setup

Overview

Architecture

Prerequisites

Installation

Docker Compose Setup

Start Monitoring Stack

Monitoring Services

Prometheus Configuration

Custom Metrics

User Collection Metrics

Cache Metrics

Celery Task Metrics

API Endpoint Metrics

Using Custom Metrics

Grafana Setup

Install Grafana

Add Prometheus Data Source

Create Dashboard

Monitoring Best Practices

Troubleshooting

Metrics Endpoint Not Found

Prometheus Not Scraping

Grafana Connection Failed

Missing Metrics

Accessing Monitoring Tools

Next Steps