Skip to content

Chapter 4: How Algorithmic Trading Works

Reading time: 18 minutes | Difficulty: Intermediate | Prerequisites: Chapters 0-3

This chapter walks through the entire pipeline -- from raw FRED data to an executed GLD order -- with real numbers from the system's actual constants and risk parameters.

The Pipeline: Eight Stages from Data to Trade

 +---------+   +----------+   +----------+   +---------+   +-----------+
 |  DATA   |-->| FEATURES |-->| STRATEGY |-->| SIGNAL  |-->| REGIME    |
 | Ingest  |   | Engine   |   | Signals  |   | Aggreg. |   | Allocator |
 +---------+   +----------+   +----------+   +---------+   +-----------+
                                                                  |
                                                                  v
 +---------+   +----------+   +----------+   +-----------+
 | SELF-   |<--| TCA      |<--| EXECUTE  |<--| COMPLIANCE|
 | LEARN   |   | (Review) |   | (Alpaca) |   | Gateway   |
 +---------+   +----------+   +----------+   +-----------+

Each stage has a single job, a defined input, and a defined output. No stage can skip another.

Stage 1: Data Ingestion

The system pulls data from five source categories every trading day:

Source What When Latency
Alpaca Live prices for 20 PM ETFs + 5 cross-asset 9:30 AM - 4:00 PM ET Real-time streaming
FRED DFII10, DGS10, DTWEXBGS, T10YIE, EFFR, VIXCLS, M2SL, WALCL 6:00 AM ET daily Previous close (T-1)
CFTC COT reports for gold and silver futures Fridays 3:30 PM ET 3-day lag (Tuesday snapshot)
LBMA AM/PM gold fix, silver fix Intraday Minutes after fix
ETF Flows GLD, IAU, SLV creation/redemption data After market close T+1

Validation: Every data point is checked against staleness thresholds. If a FRED series has not updated in 7 days (FEATURE_STALENESS_MAX_DAYS), derived features are excluded. If Alpaca streaming drops for more than 120 seconds (DEAD_MAN_SWITCH_TIMEOUT_SECONDS), the system raises an alert and holds existing positions.

Stage 2: Feature Calculation

Raw data is transformed into 100+ features. Features are the language strategies speak.

Example features from raw gold price data: - Momentum (short): 21-day return (MOMENTUM_SHORT_WINDOW) - Momentum (medium): 63-day return (MOMENTUM_MEDIUM_WINDOW) - Momentum (long): 252-day return (MOMENTUM_LONG_WINDOW) - Realized volatility: 21-day rolling annualized standard deviation - Z-score: Current price relative to 252-day mean, in standard deviations

Example features from macro data: - Real yield level: DFII10 current value - Real yield change: 5-day, 21-day, 63-day change - DXY z-score: Dollar index relative to 252-day mean - VIX percentile: Current VIX rank in trailing 252-day distribution - COT net speculative: Net long contracts by speculators, as percentile of 52-week range

Example cross-instrument features: - Gold/silver ratio: GLD price / SLV price (adjusted for share weights) - GDX/GLD ratio: Miner-to-metal spread - GLD term structure slope: If futures configured, front/back month spread as annualized percentage - Breakeven inflation: TIP/IEF ratio as proxy for T10YIE

Features are computed at 9:00 AM (pre-market, using T-1 close data) and updated continuously during market hours for intraday strategies.

Stage 3: Strategy Signal Generation

All 29 strategies consume features and output a Signal object:

Signal:
  symbol:      "GLD"
  direction:   LONG (+1) or SHORT (-1) or FLAT (0)
  strength:    0.0 to 1.0 (conviction)
  strategy_id: "real_rate_gold"
  timestamp:   2026-04-12T15:30:00-04:00

At 3:30 PM ET (REBALANCE_HOUR=15, REBALANCE_MINUTE=30), all 29 strategies produce their current signal set. Some strategies produce signals for multiple symbols. The full output is a dict mapping strategy_id to a list of Signal objects.

Categories and their signal patterns:

Category # Strategies Typical signal frequency Typical hold
Macro Regime (A) 5 Changes weekly-monthly Weeks
Curve & Carry (B) 4 Changes monthly Weeks-months
Cross-Metal Arb (C) 4 Changes weekly Days-weeks
Positioning & Flows (D) 5 Changes weekly Days-weeks
Options (E) 4 Daily (active management) Days
Microstructure (F) 4 Daily-intraday Hours-days
ML Meta (G) 2 Real-time regime gating N/A (meta)
Tail Hedge (H) 1 Always on Months

Stage 4: Signal Aggregation

The SignalAggregator combines signals from all 29 strategies into a single portfolio view. This is where the magic happens -- individual opinions become a consensus.

Weighting formula:

weight_i = inverse_vol_i * skill_score_i * capacity_headroom_i * meta_label_confidence_i

Where: - inverse_vol: Strategies with lower recent volatility get more weight (risk parity) - skill_score: How well the strategy has performed in the current regime (0 to 1) - capacity_headroom: How much of the strategy's capacity is unused (0 to 1, based on capacity_usd) - meta_label_confidence: The ML Meta Labeller's estimate of P(strategy is correct in this regime)

Output: An AggregatedSignal containing per-symbol net weights, aggregate confidence, and the list of contributing strategies.

Example aggregation: Suppose three strategies are signaling on GLD:

Strategy Direction Strength Weight
Real Rate Gold LONG 0.8 0.35
DXY Gold LONG 0.6 0.25
VIX Haven FLAT 0.0 0.15

Weighted signal for GLD: (0.8 * 0.35) + (0.6 * 0.25) + (0.0 * 0.15) = 0.43

This 0.43 becomes the raw portfolio weight for GLD before regime allocation and vol targeting.

Stage 5: Regime Allocation

The RegimeAllocator adjusts the aggregated signals based on the current market regime. Six regimes are defined:

Regime Optimizer Vol target Gold bias
Risk On Risk parity 12% (normal) Neutral
Risk Off Risk parity 12% (normal) Long bias
Inflation Risk parity 14% (expanded) Strong long
Deflation Min-variance 10% (reduced) Neutral-long
Crisis Min-variance 8% (defensive) Strong long
Transition Min-variance 10% (cautious) Neutral

The Regime Classifier (S1.G.02) uses HMM + XGBoost on VIX, DFII10, DXY, credit spreads, and yield curve slope to detect the current regime. The allocator then:

  1. Selects the optimization method (risk parity or min-variance)
  2. Sets the vol target (10-18% annualized band, DEFAULT_VOL_TARGET = 0.12)
  3. Scales all positions to hit the target ex-ante volatility
  4. Applies drawdown-responsive deleveraging if drawdown > 10% (soft cap)

Vol targeting example: If the aggregator outputs 0.43 weight for GLD, and GLD's trailing 21-day vol is 16%, and the target portfolio vol is 12%: 

vol_scalar = target_vol / position_vol = 0.12 / 0.16 = 0.75
adjusted_weight = 0.43 * 0.75 = 0.32

The final GLD weight after vol targeting is 0.32 (32% of NAV).

Stage 6: Compliance Gateway

Before any order reaches the broker, it passes through four independent compliance checks:

Check 1: Wash-Sale Detector

IRS wash-sale rule: if you sold a security at a loss within the past 30 days, you cannot repurchase the same or substantially identical security without losing the tax deduction on the loss. The system tracks all realized losses and blocks repurchases within the 30-day window.

Check 2: Reg-T / Pattern Day Trader

FINRA Rule 4210: accounts with fewer than $25K equity are limited to 3 day trades per rolling 5-business-day window. The system counts round trips and blocks orders that would trigger PDT classification.

Check 3: Position Limits

Three hard limits from constants.py: - Single name: No one symbol can exceed 15% of NAV (DEFAULT_MAX_SINGLE_NAME_PCT = 0.15) - Sector: No one sector can exceed 40% of NAV (DEFAULT_MAX_SECTOR_PCT = 0.40) - Gross leverage: Total gross exposure cannot exceed 2.0x NAV (DEFAULT_MAX_GROSS_LEVERAGE = 2.0)

Check 4: Restricted-List Filter

Hard block on any symbol flagged as restricted (e.g., due to corporate action, trading halt, or regulatory issue). Currently no PM instruments are restricted, but the check runs every cycle.

If any check fails: The order is BLOCKED. An AuditEventType entry is logged with the correlation ID, the blocking check, and the specific violation. The strategy's signal remains recorded but the order does not reach the broker.

Stage 7: Execution (Alpaca)

Approved orders are routed to Alpaca based on order size relative to the instrument's ADV:

Condition Execution algorithm Slicing
Order < 1% of ADV Market order Single fill
1-5% of ADV VWAP 10 slices (VWAP_DEFAULT_NUM_SLICES)
> 5% of ADV TWAP with participation cap 10 slices, max 10% of bar volume

Slippage estimates: - Liquid (GLD, SLV, GDX, IAU): 2 bps (MIN_SLIPPAGE_LIQUID_BPS) - Illiquid (PALL, SLVP, RING): 10 bps (MIN_SLIPPAGE_ILLIQUID_BPS) - Commission: 5 bps assumed (DEFAULT_COMMISSION_BPS)

Reconciliation: Every 60 seconds (RECONCILIATION_INTERVAL_SECONDS), the system pulls position data from Alpaca and compares it against the internal position book. Any discrepancy triggers an alert. The broker's state is treated as the source of truth.

Stage 8: Kill Switch and Post-Trade

Kill Switch: Four Escalating Tiers

The kill switch monitors portfolio health continuously and triggers at four severity levels:

 NORMAL ──> WARN ──> THROTTLE ──> NO_NEW ──> FLATTEN
   |          |          |            |           |
   OK      Alert     Reduce size   Hold only   Close ALL
           sent      by 50%       no new       positions
                                  trades       immediately
Tier Trigger Action
WARN Daily loss > -1.5% or drawdown > -8% Alert to dashboard and notifications. All strategies continue.
THROTTLE Daily loss > -2% or drawdown > -10% New positions sized at 50%. Existing positions held.
NO_NEW Daily loss approaching -3% or drawdown > -12% No new positions allowed. Only exits permitted.
FLATTEN Daily loss = -3% (DEFAULT_CIRCUIT_BREAKER_DAILY_LOSS) or weekly loss = -5% (DEFAULT_CIRCUIT_BREAKER_WEEKLY_LOSS) or drawdown = -15% (DEFAULT_MAX_DRAWDOWN_PCT) All positions closed immediately. System goes to 100% cash. Manual restart required.

TCA (Transaction Cost Analysis)

After every trade, the system records: - Fill price vs. decision price (slippage) - Fill price vs. VWAP (implementation shortfall) - Time to fill (latency) - Market impact (price move between order entry and fill)

TCA data feeds back into the self-learning loop to improve execution algorithm selection and slippage estimates.

Worked Example: Real Rate Gold Signals LONG

Let us walk a single trade through all 8 stages with concrete numbers.

Stage 1: Data

At 6:00 AM, the system fetches FRED data. DFII10 (10Y real yield) has dropped from +1.8% to +1.2% over the past 21 days.

Stage 2: Features

The feature engine computes: - DFII10 level: 1.2% (below the strategy's scale-in threshold of 1.5%) - DFII10 21-day change: -0.60% (significant decline) - DFII10 z-score on 252-day window: -1.3 (falling toward lower range)

Stage 3: Signal

The Real Rate Gold strategy (S1.A.01) produces: 

Signal(symbol="GLD", direction=LONG, strength=0.7, strategy_id="real_rate_gold")
Strength is 0.7 (not 1.0) because real yields are falling but not yet negative.

Stage 4: Aggregation

Other strategies are also signaling on GLD: - DXY Gold: LONG 0.5 (dollar weakening modestly) - VIX Haven: FLAT (VIX at 18, not elevated) - Central Bank Gold: LONG 0.6 (central banks are net buyers)

Aggregated weight for GLD: 0.45 (weighted average of all contributing signals)

Stage 5: Regime Allocation

Regime Classifier output: Risk Off (real yields falling, dollar weakening, credit spreads stable). - Optimizer: risk parity - Vol target: 12% - Gold bias: long

GLD trailing 21-day vol: 15% Vol-targeted weight: 0.45 * (0.12 / 0.15) = 0.36

After regime adjustment (risk-off gold bias +10%): 0.40

Stage 6: Compliance

With NAV of $1,000,000, the GLD position would be \(400,000 (40% of NAV). - Single-name check: 40% > 15% limit -- **EXCEEDS LIMIT.** - Compliance trims the position to 15% of NAV: **\)150,000. - Remaining 25% is redistributed across other gold instruments (IAU, SGOL) by the vehicle router. - Sector check: Total gold ETF exposure (GLD + IAU + SGOL) = ~35% < 40% sector limit. PASS. - Gross leverage: 35% < 200%. PASS. - Wash sale: No recent GLD sales at a loss. PASS. - Reg-T: No PDT concern. PASS.**

Stage 7: Execution

Orders: - Buy $150,000 GLD (~65 shares at $230/share). Market order (< 1% of 8M ADV). - Buy $120,000 IAU (~2,400 shares at $50/share). Market order. - Buy $80,000 SGOL (~1,500 shares at $53/share). Limit order (> 1% of 2M ADV, but under VWAP threshold).

All fills within 2 bps of decision price. Total slippage: ~$7. Commission: $0 (Alpaca).

Stage 8: Post-Trade

  • Reconciliation confirms: positions match broker state.
  • TCA logged: slippage 1.8 bps (within expected range).
  • Kill switch status: NORMAL (daily P&L flat at this point).
  • Self-learning: no drift detected, no regime change.

Total elapsed time from signal generation to fill: approximately 45 seconds.

The Self-Learning Loop

After every trading day, the system runs a closed-loop improvement process:

 4:30 PM  +-- Drift Detection
          |   KS test + PSI on all features vs training distribution
          |   Threshold: PSI > 0.2 or KS p-value < 0.01
          |
 5:00 PM  +-- Loss-Day Postmortem (if daily P&L < -1% of NAV)
          |   Automated analysis: which strategies contributed to loss?
          |   Was it signal failure or execution failure?
          |
 5:30 PM  +-- Decay Monitor Check
          |   Rolling 6-month Sharpe < 60% of in-sample -> WARN
          |   Rolling 6-month Sharpe < 50% of in-sample -> retrain
          |   Rolling 24-month Sharpe < 30% of in-sample -> QUARANTINE
          |
10:00 PM  +-- Retrain Cycle (if triggered)
          |   1. Retrain model on latest data (2-year train window)
          |   2. Walk-forward validation (1-quarter test, 1-month embargo)
          |   3. Deploy as SHADOW (paper-trades alongside production)
          |   4. If shadow beats production over evaluation window -> PROMOTE
          |   5. Old model -> ARCHIVED

Model lifecycle: STAGING (newly trained) --> PRODUCTION (live trading) --> ARCHIVED (replaced by better version)

The retrain uses walk-forward validation with the constants from constants.py: - Train window: 504 bars (~2 years, WALK_FORWARD_DEFAULT_TRAIN_WINDOW) - Test window: 63 bars (~1 quarter, WALK_FORWARD_DEFAULT_TEST_WINDOW) - Embargo: 21 bars (~1 month, WALK_FORWARD_DEFAULT_EMBARGO)

Probability of Backtest Overfitting (PBO): Rejected if PBO > 50% (PBO_REJECT_THRESHOLD). Deflated Sharpe Ratio (DSR): Required confidence > 95% (DSR_ACCEPT_CONFIDENCE).

For OpB: The system is a pipeline. Data flows in, math processes it through 8 stages, trades come out. The kill switch and compliance gateway are safety nets that cannot be overridden by the strategies. If you see NORMAL on the kill switch and "Synced" on reconciliation, the system is doing its job.

For OpA: The key architectural choice is the separation between signal aggregation (conviction-weighted) and regime allocation (risk-budget-weighted). This two-stage process means a high-conviction signal can still get sized down in a crisis regime. The compliance gateway operates as an independent constraint layer that cannot be bypassed by the allocator.

Next: Chapter 5: Strategies Explained -- all 29 strategies, organized by category, with when they work and when they break.