Reconciling Heterogeneous Data Sources

Several of my systems live or die on data reconciliation: pulling from many sources that disagree on schema, freshness and rate limits, then presenting one coherent answer.

Patterns that hold up

1. Isolate the ingestion store. In a crypto-analytics platform, external provider data lands in its own dataset so a slow source never contends with core reads.
2. Centralize the clients. One rate-limit-aware client layer per provider beats scattering retry logic across the codebase.
3. Make the derived value reproducible. A transparency score or a financial balance must be recomputable from an audit trail — otherwise you can't defend it when a customer disputes it.

Financial reconciliation is a special case

When two sources represent money — e.g. ad spend reported by a platform vs the bank debit — you need point-in-time snapshots. In an adtech / CRM platform I used T0/T1/T2 balance projections so async reconciliation never left the ledger in an ambiguous state.

The through-line: treat every external number as eventually-consistent and every derived number as auditable.