Signals Outage Runbook

Use this runbook when signal generation or downstream delivery looks stale.

Detection

• GET /api/v1/signals/latest fails or returns obviously stale payloads
• GET /api/v1/signals/delivery-status shows delivery channels misconfigured
• daemon telemetry is stale
• Discord/Telegram mirrors stop posting when those channels are enabled

First Checks

1. API health:

curl -sS https://api.qgtmai.com/health

1. Signal generation:

curl -sS https://api.qgtmai.com/api/v1/signals/latest

1. Delivery configuration:

curl -sS https://api.qgtmai.com/api/v1/signals/delivery-status

1. Daemon state:

curl -sS https://api.qgtmai.com/api/v1/daemon/telemetry

Host-Level Checks

On the production droplet:

sudo systemctl status qgtm-api qgtm-daemon qgtm-redis
sudo journalctl -u qgtm-daemon -n 100 --no-pager
sudo journalctl -u qgtm-api -n 100 --no-pager

Common Causes

Symptom	Likely Cause	Action
No fresh signals	Daemon stopped or heartbeat stale	Restore qgtm-daemon, then verify telemetry
Signals compute locally but not externally	API degraded or Redis unhealthy	Check /health, Redis, and API logs
Delivery status shows unconfigured channels	Discord or Telegram credentials missing	Fix runtime config before expecting external delivery
Delivery lag with healthy daemon	Consumer/config issue in signal publishing layer	Check qgtm_signals and bot logs/config

Communication

If external delivery is actually enabled for users, post a brief status update in the active delivery channel:

Signal delivery is delayed. The trading/runtime stack is under review and we
will post the next update within 30 minutes.

Recovery

1. Restore API, daemon, and Redis health
2. Verify signals/latest, daemon/telemetry, and delivery-status
3. Replay or reissue missed downstream notifications only if the delivery layer was enabled and the operator confirms they should be resent
4. Write a short incident note with root cause and follow-up action