Broker Outage Runbook
Detection
- Alpaca API returns 5xx or times out
- WebSocket disconnects and fails to reconnect after 3 attempts
- Order submission fails with connection error
- Position reconciliation shows stale data
Response
- Check Alpaca status page: https://status.alpaca.markets
- Check your Alpaca dashboard manually for position state
- Do NOT submit new orders until connectivity is confirmed
- Log the outage start time
If Outage > 15 Minutes
- Cancel all pending orders (they may be queued)
- Record current positions from last known good state
- Switch to read-only mode
- Notify team via Discord/Telegram
Recovery
- Verify API connectivity with health check
- Reconcile positions: broker state vs local state
- Resolve any discrepancies (partial fills during outage)
- Resume trading with reduced size for first hour
- Log outage duration and impact