Chapter 7: Our Data Edge
The #1 rule: Garbage in, garbage out. The best strategy in the world is worthless if fed stale data, looked-ahead data, or data with survivorship bias. Our data pipeline is built for correctness first, speed second.
The 10 Data Pillars
We consume data from 10 distinct sources. Each one feeds specific strategies and has its own quirks, latencies, and failure modes.
PILLAR SOURCE STRATEGIES FED UPDATE
====== ======================== =============== ================
1 LBMA Fix + Vault Holdings F2, D4 2x daily + monthly
2 COMEX Warehouse Stocks D4, B4 Daily
3 CFTC COT D1, D2 Weekly (Fri)
4 ETF Flows (GLD/SLV) D3 Daily
5 Central Bank Reserves A5 Quarterly
6 Refinery Premiums C1, D5 Daily-weekly
7 Mint Sales D5 (demand proxy) Monthly
8 FRED Macro A1-A4, G2 Daily
9 Alt Data G1 (features) Varies
10 PIT Enforcement Layer ALL (validation) Always-on
Pillar 1: LBMA Fix Prices + Vault Holdings
What We Get
The London Bullion Market Association sets the global benchmark price for gold twice daily and silver once daily.
LBMA GOLD FIX:
Morning fix: 10:30 AM London time (5:30 AM ET)
Afternoon fix: 3:00 PM London time (10:00 AM ET)
LBMA SILVER FIX:
Once daily: 12:00 PM London time (7:00 AM ET)
Why it matters: The fix price is the reference for physical gold transactions globally. Central banks, miners, refiners, and jewellers all settle against it. When futures diverge from the fix by more than the arbitrage band, our F2 (Fix Dislocation) strategy trades the convergence.
LBMA Vault Holdings
Monthly data on total gold and silver held in London vaults. This is the physical backbone of the global OTC gold market.
London Vault Gold Holdings (as of Q4 2024): ~8,500 tonnes
Monthly change matters more than level:
- Declining holdings = physical metal leaving London -> bullish signal
- Rising holdings = metal flowing in -> neutral/slightly bearish
Pillar 2: COMEX Warehouse Stocks
Registered vs Eligible
COMEX warehouses hold gold and silver in two categories:
REGISTERED: Metal that has been warranted and is available for
futures delivery. This is the "deliverable supply."
ELIGIBLE: Metal stored in COMEX vaults but NOT warranted for
delivery. The owner has not made it available.
TOTAL = REGISTERED + ELIGIBLE
THE KEY METRIC: Registered stock changes.
A drop in registered = metal being pulled for delivery = physical demand.
Strategy connection: Our D4 (COMEX Warehouse Drawdown) strategy monitors registered stock changes. When registered gold drops >5% in a week, it signals physical tightness.
EXAMPLE (Feb 2024):
Registered gold: Week 1: 8.2M oz
Week 2: 7.6M oz (-7.3% weekly change)
Signal: LONG gold (physical tightness)
Result: Gold rallied 4.2% over next 3 weeks
Pillar 3: CFTC Commitments of Traders (COT)
The Report Structure
Every Tuesday the CFTC snapshots futures positions. Published Friday at 3:30 PM ET.
GOLD COT REPORT (Disaggregated Format):
================================================================
CATEGORY LONG SHORT NET
================================================================
Managed Money 285,000 42,000 +243,000
(hedge funds, CTAs)
Commercials 78,000 312,000 -234,000
(miners, refiners)
Swap Dealers 95,000 110,000 -15,000
(banks making markets)
Other Reportable 45,000 38,000 +7,000
(smaller institutions)
Non-Reportable 22,000 23,000 -1,000
(retail)
================================================================
Which Category We Trade
Managed Money is our primary signal. These are the trend followers and macro funds. When their net position hits 2-sigma extremes, the trade is crowded.
Commercials are our secondary signal. They are natural hedgers (miners sell forward). When commercial hedging pressure is extreme, it signals the risk premium is rich (D2 strategy).
PIT Enforcement for COT
COT DATA TIMING:
Tuesday: Position snapshot taken (market close)
Friday: Data published at 3:30 PM ET
Our use: EARLIEST allowed date is Saturday (next trading day: Monday)
THIS IS NON-NEGOTIABLE.
In backtests, we enforce a point_in_time_offset of 3 calendar days:
- Tuesday snapshot -> available for signals starting MONDAY
- Using it on Friday (publication day) is already aggressive
- Using it on Thursday or earlier is LOOKAHEAD BIAS
Backtest Sharpe with correct PIT: 0.62
Backtest Sharpe with 1-day cheat: 0.91 (+47% inflation!)
The difference is fake. The 0.91 is a lie.
Pillar 4: ETF Flows (GLD/SLV Creations/Redemptions)
How ETF Creation/Redemption Works
CREATION (bullish signal):
1. Authorized Participant (AP) buys physical gold on open market
2. AP delivers gold to GLD trust (HSBC vault in London)
3. Trust issues new GLD shares to AP
4. AP sells shares on exchange
Result: More GLD shares outstanding = institutional BUYING
REDEMPTION (bearish signal):
1. AP buys GLD shares on exchange
2. AP delivers shares to trust
3. Trust releases physical gold to AP
4. AP sells gold on open market
Result: Fewer GLD shares outstanding = institutional SELLING
What we track:
daily_flow = shares_outstanding_today - shares_outstanding_yesterday
flow_z = zscore(rolling_5d_flow, window=252)
Positive flow_z = net creation = buying pressure
Negative flow_z = net redemption = selling pressure
Key insight: Flows lead price in gold ETFs by 1-3 days because the creation/redemption process takes time to execute and the AP decision happens before the price move.
Pillar 5: Central Bank Reserves
Data Sources
SOURCE FREQUENCY COUNTRIES DELAY
======================== =========== ========== ===========
IMF IFS (International Monthly ~190 2-3 months
Financial Statistics)
Individual CB disclosures:
- PBOC (China) Monthly 1 1 month
- RBI (India) Weekly 1 1 week
- CBRT (Turkey) Weekly 1 1 week
- NBP (Poland) Monthly 1 1 month
- MAS (Singapore) Monthly 1 1 month
(+ 15 more tracked CMs)
WGC (World Gold Council) Quarterly Top 20 1 month
demand report
The 20 Countries We Track
RANK COUNTRY TONNES % OF RESERVES TREND (2022-2024)
==== ============= ====== ============= =================
1 USA 8,133 71.3% Stable
2 Germany 3,352 69.9% Stable
3 Italy 2,452 65.5% Stable
4 France 2,437 67.2% Stable
5 Russia 2,333 28.1% Buying
6 China 2,264 4.9% AGGRESSIVE buying
7 Switzerland 1,040 6.5% Stable
8 India 854 10.2% Buying
9 Japan 846 4.3% Stable
10 Turkey 585 34.2% Buying
...
20 Singapore 225 3.8% Buying
China is the most important. PBOC has been buying 20-30 tonnes/month since 2022. At $2,300/oz, that is $1.5-2.2B of monthly demand from a single buyer. This is structural, not tactical, and it provides a floor under gold.
Pillar 6: Refinery Premiums (Shanghai/London/India)
What Premiums Tell Us
A "premium" means the local price exceeds the international benchmark. A "discount" means the opposite.
Shanghai Premium = SGE gold price - LBMA gold price (in USD/oz)
Normal range: +$5 to +$15/oz (Chinese demand is steady)
Bullish signal: > +$30/oz (Chinese demand surging)
Bearish signal: < -$5/oz (Chinese dumping or import controls)
India Premium = Mumbai gold price - LBMA gold price (in USD/oz)
Normal range: +$2 to +$10/oz (steady jewellery demand)
Festival spike: +$15 to +$40/oz (Diwali, Akshaya Tritiya)
Discount: < $0/oz (demand destruction from high prices)
The spread between Shanghai and London is the single best real-time indicator of Chinese physical demand.
Pillar 7: Mint Sales
Monthly sales data from three major government mints:
MINT PRODUCT TYPICAL MONTHLY DATA RELEASE
================ ================ ================= ============
US Mint Gold Eagles 50,000-150,000 oz 1st week
Silver Eagles 1-3M oz 1st week
Perth Mint Gold bars/coins 30,000-80,000 oz Mid-month
Royal Canadian Gold Maple Leafs 20,000-60,000 oz Mid-month
Mint (RCM)
Mint sales are a proxy for retail and small institutional demand. Spikes in US Eagle sales preceded gold rallies in 2020 and 2024.
Pillar 8: FRED Macro Data
The FRED Series We Pull
SERIES NAME UPDATE STRATEGIES
========= ============================ ======== ==========
DFII10 10Y TIPS Real Yield Daily A1
DGS10 10Y Treasury Nominal Yield Daily A1, A3
DXY US Dollar Index (via proxy) Real-time A1, A2
T10YIE 10Y Breakeven Inflation Daily A3
VIXCLS VIX Close Daily A4, E1-E4
M2SL M2 Money Supply Weekly A3
WALCL Fed Balance Sheet (Total) Weekly A3
FEDFUNDS Fed Funds Effective Rate Daily A1
BAMLH0A0 ICE BofA HY OAS Daily A4
DTWEXBGS Trade-Weighted Dollar Daily A2
Release Schedule for Key Series
DAILY (available by 6:00 PM ET):
DFII10, DGS10, T10YIE, VIXCLS, FEDFUNDS, BAMLH0A0
WEEKLY (Thursday):
WALCL (Fed balance sheet)
MONTHLY (~2 weeks after month-end):
M2SL
8X/YEAR (FOMC decisions):
Rate decisions, dot plots, SEP
Pillar 9: Alt Data
Alternative data supplements our core pillars with non-traditional signals.
SOURCE WHAT STRATEGY USE
=================== ============================ =================
Vessel tracking Gold shipments via tanker Physical flow proxy
(AIS data) routes (Switzerland->China, (feeds G1 features)
UK->India)
Weather data Temperature forecasts for Silver industrial
(NOAA) solar panel production demand proxy
regions
Satellite imagery Mine activity levels at Supply-side signal
major gold/silver mines for medium-term
(Goldstrike, Cripple Creek)
News sentiment NLP sentiment scoring on G1 feature input
(GDELT, news APIs) gold/silver related articles (contrarian indicator
Real-time media mentions at extremes)
Supply chain data Semiconductor fab orders Silver industrial
(silver is used in demand leading
chip manufacturing) indicator
Alt data is fed to the G1 Meta-Labeller as features. None of it generates standalone signals -- it improves the ensemble.
Pillar 10: Point-in-Time (PIT) Enforcement
Why PIT Matters More Than Anything Else
Point-in-time integrity means every signal in our backtest uses ONLY data that was available at the moment the signal would have been generated in real time. Violating this is called "lookahead bias" and it makes backtests worthless.
EXAMPLE OF LOOKAHEAD BIAS:
BAD (lookahead):
- Monday signal uses COT data that won't be published until Friday
- Monthly signal uses WASDE data at 11:55 AM when it releases at 12:00 PM
- Feature uses revised GDP when only the advance estimate was available
GOOD (PIT-compliant):
- Monday signal uses LAST Friday's COT data
- WASDE signal is generated AFTER 12:00 PM release on report day
- Feature uses advance GDP estimate, updates when revision releases
How We Enforce PIT
EVERY data source has a PIT timestamp:
class DataPoint:
value: float
as_of: datetime # When the data was MEASURED
available_at: datetime # When we could FIRST USE it
source: str
The backtest engine checks:
IF signal_timestamp < data_point.available_at:
RAISE LookaheadViolation(
f"Signal at {signal_timestamp} uses data not available until "
f"{data_point.available_at}"
)
PIT Delays by Source
SOURCE AS_OF AVAILABLE_AT DELAY
============== ================ ================ ========
LBMA Fix Fix time Fix time + 1 min 1 min
COMEX Stocks End of day Next day 10 AM ET ~14 hours
CFTC COT Tuesday close Friday 3:30 PM ET 3.5 days
GLD Flows End of day Next day 8 AM ET ~14 hours
IMF IFS Month-end 2-3 months later 60-90 days
PBOC Gold Month-end Next month 7th 7-37 days
FRED Daily 4 PM ET 6 PM ET same day 2 hours
FRED Weekly Week-end Thursday release 3-6 days
US Mint Sales Month-end ~5th of next month 5-35 days
Alt (satellite) Image date Processing + 1 day 1-7 days
The Cost of Getting PIT Wrong
STRATEGY SHARPE (CORRECT PIT) SHARPE (1-DAY CHEAT) INFLATION
============== ==================== ==================== =========
COT Extreme 0.62 0.91 +47%
Real Rate Gold 0.71 0.84 +18%
ETF Flows 0.55 0.78 +42%
COMEX Drawdown 0.41 0.63 +54%
Meta-Labeller 0.89 1.31 +47%
AVERAGE SHARPE INFLATION FROM 1-DAY LOOKAHEAD: +40%
This means almost HALF the reported performance of many academic
papers and commercial systems is fake -- caused by sloppy PIT handling.
Our CI pipeline runs a PIT audit on every backtest. If any signal uses data before its available_at timestamp, the test fails and the PR is blocked.
Data Quality Pipeline
Every data point passes through 8 quality gates before it touches a strategy:
┌─── DATA QUALITY GATES ──────────────────────────────────────────┐
│ │
│ 1. FRESHNESS: Is this from the expected time period? │
│ Stale LBMA fix from yesterday used as today's = fatal. │
│ │
│ 2. COMPLETENESS: All fields present? No nulls? │
│ Missing silver fix = skip that day's fix signals. │
│ │
│ 3. RANGE: Is gold between $500 and $10,000/oz? │
│ Is silver between $5 and $200/oz? │
│ Outside range = reject and alert. │
│ │
│ 4. CONSISTENCY: Gold > silver always (in $/oz). │
│ Gold/silver ratio between 15 and 200. │
│ High >= Low on every bar. │
│ │
│ 5. PIT COMPLIANCE: available_at >= signal_timestamp? │
│ Enforced in backtest. Logged in live. │
│ │
│ 6. CROSS-REFERENCE: LBMA fix vs COMEX close within $5? │
│ GLD NAV vs gold spot within 0.5%? │
│ Discrepancy = flag for investigation. │
│ │
│ 7. TIMEZONE: All timestamps stored in UTC. │
│ Conversion to ET/London only for display. │
│ Mixing timezones causes off-by-one-day errors. │
│ │
│ 8. REVISION HANDLING: Use FIRST available version. │
│ GDP advance, not final. COT as-reported, not revised. │
│ Revisions are new data points, not overwrites. │
│ │
└──────────────────────────────────────────────────────────────────┘
Summary
┌─── KEY TAKEAWAYS ───────────────────────────────────────────────┐
│ │
│ 1. 10 data pillars spanning physical, futures, macro, flows, │
│ and alternative data │
│ │
│ 2. PIT enforcement is the single most important data practice. │
│ A 1-day lookahead inflates Sharpe by 40% on average. │
│ Our CI blocks any backtest that violates PIT. │
│ │
│ 3. Government data (FRED, CFTC) is free. LBMA and COMEX data │
│ is low-cost. Our total data spend is under $500/month. │
│ │
│ 4. Physical market data (LBMA vaults, COMEX stocks, SGE, │
│ refinery premiums) gives us an edge over funds that only │
│ trade paper gold. │
│ │
│ 5. Every data point has an as_of timestamp and an available_at │
│ timestamp. The gap between them is the PIT delay. Respect │
│ it or your backtest is fiction. │
│ │
└──────────────────────────────────────────────────────────────────┘
Next up: How all this code fits together. Chapter 8: System Architecture -->