Ga naar hoofdinhoud

Monitoring

Server monitoring en alerting setup.

Monitoring Stack​

┌─────────────────────────────────────────────────────────────┐
│                     Monitoring                               │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐        │
│  │ Uptime Kuma │  │   Grafana   │  │   Loki      │        │
│  │  (Health)   │  │  (Metrics)  │  │   (Logs)    │        │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘        │
│         │                │                │                │
│         └────────────────┼────────────────┘                │
│                          │                                  │
│                    ┌─────▼─────┐                           │
│                    │  Alerts   │                           │
│                    │ (Discord) │                           │
│                    └───────────┘                           │
└─────────────────────────────────────────────────────────────┘
                          │
         ┌────────────────┼────────────────┐
         │                │                │
         ▼                ▼                ▼
   ┌───────────┐   ┌───────────┐   ┌───────────┐
   │SpacetimeDB│   │   Nginx   │   │    R2     │
   └───────────┘   └───────────┘   └───────────┘

Uptime Kuma​

Simpele uptime monitoring.

Installatie​

# Docker
docker run -d \
    --name uptime-kuma \
    -p 3001:3001 \
    -v uptime-kuma:/app/data \
    louislam/uptime-kuma:1

Monitors Configureren​

  1. SpacetimeDB WebSocket

    • Type: TCP Port
    • Hostname: localhost
    • Port: 3000
    • Interval: 60s

  2. HTTP Health Check

  3. Admin Panel

Discord Alerts​

  1. Create Discord webhook in server settings
  2. In Uptime Kuma: Settings → Notifications → Add → Discord
  3. Paste webhook URL
  4. Assign to monitors

SpacetimeDB Metrics​

Ingebouwde Metrics​

# Via CLI
spacetime metrics milenas-treehouse

Output:

  • Connected clients
  • Reducer calls/sec
  • Database size
  • Memory usage

Custom Metrics Endpoint​

// In lib.rs - custom health endpoint
#[reducer]
pub fn get_server_stats(ctx: &ReducerContext) -> Result<String, String> {
    if !is_admin(ctx) {
        return Err("Admin only".to_string());
    }

    let stats = serde_json::json!({
        "online_players": ctx.db.player_online().iter().count(),
        "total_players": ctx.db.player().iter().count(),
        "timestamp": ctx.timestamp,
    });

    Ok(stats.to_string())
}

Log Management​

Loki + Promtail​

# docker-compose.yml
services:
  loki:
    image: grafana/loki:2.9.0
    ports:
      - "3100:3100"
    volumes:
      - loki-data:/loki

  promtail:
    image: grafana/promtail:2.9.0
    volumes:
      - /var/log:/var/log
      - ./promtail-config.yml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml

Promtail Config​

# promtail-config.yml
server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: spacetimedb
    static_configs:
      - targets:
          - localhost
        labels:
          job: spacetimedb
          __path__: /var/log/spacetimedb/*.log

Grafana Dashboard​

Installatie​

docker run -d \
    --name grafana \
    -p 3002:3000 \
    -v grafana-data:/var/lib/grafana \
    grafana/grafana

Dashboard JSON​

{
    "title": "Milena's Treehouse",
    "panels": [
        {
            "title": "Online Players",
            "type": "stat",
            "datasource": "SpacetimeDB",
            "query": "SELECT COUNT(*) FROM player_online"
        },
        {
            "title": "Players Over Time",
            "type": "timeseries",
            "datasource": "SpacetimeDB",
            "query": "..."
        }
    ]
}

Alerts​

Alert Regels​

AlertConditieActie
Server DownNo response 2minDiscord + SMS
High Memory>80% RAMDiscord
Slow Queries>1s responseDiscord
Error Spike>10 errors/minDiscord

Discord Webhook Script​

#!/bin/bash
# alert.sh

WEBHOOK_URL="https://discord.com/api/webhooks/..."

send_alert() {
    local title="$1"
    local message="$2"
    local color="$3"

    curl -X POST "$WEBHOOK_URL" \
        -H "Content-Type: application/json" \
        -d "{
            \"embeds\": [{
                \"title\": \"$title\",
                \"description\": \"$message\",
                \"color\": $color
            }]
        }"
}

# Usage
send_alert "Server Alert" "SpacetimeDB is down!" 16711680

Health Check Script​

#!/bin/bash
# healthcheck.sh

SPACETIME_HOST="localhost:3000"
DISCORD_WEBHOOK="..."

check_spacetime() {
    nc -z localhost 3000
    return $?
}

if ! check_spacetime; then
    ./alert.sh "SpacetimeDB Down" "Server is not responding" 16711680

    # Attempt restart
    sudo systemctl restart spacetimedb

    sleep 30

    if check_spacetime; then
        ./alert.sh "SpacetimeDB Recovered" "Server is back online" 65280
    fi
fi

Cron Job​

# Check every minute
* * * * * /home/spacetime/healthcheck.sh

Daily Report​

#!/bin/bash
# daily_report.sh

WEBHOOK_URL="..."

# Get stats
ONLINE=$(spacetime sql milenas-treehouse "SELECT COUNT(*) FROM player_online")
TOTAL=$(spacetime sql milenas-treehouse "SELECT COUNT(*) FROM player")
NEW_TODAY=$(spacetime sql milenas-treehouse "SELECT COUNT(*) FROM player WHERE created_at > datetime('now', '-1 day')")

curl -X POST "$WEBHOOK_URL" \
    -H "Content-Type: application/json" \
    -d "{
        \"embeds\": [{
            \"title\": \"Daily Report\",
            \"fields\": [
                {\"name\": \"Online Now\", \"value\": \"$ONLINE\", \"inline\": true},
                {\"name\": \"Total Players\", \"value\": \"$TOTAL\", \"inline\": true},
                {\"name\": \"New Today\", \"value\": \"$NEW_TODAY\", \"inline\": true}
            ],
            \"color\": 3447003
        }]
    }"

Best Practices​

  1. Monitor alles - WebSocket, HTTP, database
  2. Alert slim - Niet te veel, niet te weinig
  3. Log centraal - Alles naar Loki
  4. Daily reports - Overzicht houden
  5. Backups testen - Regelmatig restore test

Volgende​