Skip to content

Broker Outage Runbook

Detection

  • Alpaca API returns 5xx or times out
  • WebSocket disconnects and fails to reconnect after 3 attempts
  • Order submission fails with connection error
  • Position reconciliation shows stale data

Response

Immediate (< 5 minutes)

  1. Check Alpaca status page: https://status.alpaca.markets
  2. Check your Alpaca dashboard manually for position state
  3. Do NOT submit new orders until connectivity is confirmed
  4. Log the outage start time

If Outage > 15 Minutes

  1. Cancel all pending orders (they may be queued)
  2. Record current positions from last known good state
  3. Switch to read-only mode
  4. Notify team via Discord/Telegram

Recovery

  1. Verify API connectivity with health check
  2. Reconcile positions: broker state vs local state
  3. Resolve any discrepancies (partial fills during outage)
  4. Resume trading with reduced size for first hour
  5. Log outage duration and impact