Data zones: raw → clean → view (design)#

Design note for the WRDS-scale data lifecycle in numeraire-dataset. Written before the implementation (W3-1). The three-zone split makes preprocessing reproducible and pinnable: a numeraire result’s data_vintage string must trace back, unambiguously, to exactly the bytes and the exact transform recipe that produced its inputs.

Why three zones#

Preprocessing is part of the method. A Sharpe or an alpha is only reproducible if the cleaning — delisting adjustments, share-code filters, CCM link windows, the accounting-to-return lag — is pinned as tightly as the model. So the pipeline is split into three zones with hard boundaries:

raw/     immutable download cache, keyed by (source, vintage)      — bytes as pulled, never edited
clean/   deterministic transforms of raw → tidy PIT tables         — a recipe hash pins the transform
view/    numeraire TimeSeriesView / CrossSectionView builders      — lazy numeraire import, no new state

Arrows point one way: view reads clean, clean reads raw, raw reads the outside world. Nothing downstream mutates an upstream zone.

raw — immutable, pull-stamped#

  • One directory per (source, vintage), e.g. raw/crsp_msf/2024-12/. vintage is the pull’s identity: a WRDS query date, a FRED-MD month, a Ken French release. Content is the bytes exactly as pulled (parquet/csv), never edited in place — a re-pull with different content is a new vintage directory, not an overwrite.

  • Each pull writes a sibling _meta.json: {source, vintage, pulled_at, query_hash, pit_status, row_count, content_digest}. pit_status records whether the pull is itself point-in-time (a dated vintage) or a latest-snapshot convenience pull (revised, not PIT) — so a downstream golden can refuse a non-PIT source.

  • The cache directory is user-configurable and defaults OUTSIDE the repo ($NUMERAIRE_DATA → platform cache dir, per paths.data_home). Licensed data (CRSP/Compustat) lives only here, on the user’s machine behind their own WRDS credentials — never in any git history.

clean — deterministic, recipe-hashed#

  • A clean table is produced by one or more steps: pure frame(s) → frame functions with all parameters explicitly declared (no hidden globals, no ambient config). A step is deterministic: same inputs + same params → same output, bit-for-bit.

  • Every step’s identity is (name, version, params). Its recipe hash is the SHA-256 of the canonical JSON of {name, version, params, inputs: [<recipe hash of each input>]}. The hash therefore chains: a clean table’s recipe hash transitively pins every upstream transform and every raw vintage that fed it. Bump version when a step’s logic changes in a way that should invalidate downstream caches.

  • Steps register in an open registry (the same pattern as numeraire’s evaluator registry — register_step / get_step / available_steps, a module-global dict, KeyError on a duplicate name unless overwrite). No second registration style is invented. A @step(...) decorator is sugar over register_step.

view — numeraire builders, lazy#

  • view builders turn a clean table into a numeraire TimeSeriesView / CrossSectionView. They import numeraire lazily (as sources.to_timeseries_view already does), so the clean + raw zones stay numeraire-free and installable without it. view adds no persisted state.

data.lock.json#

A single lock file records, for each built clean (and pinned raw) artifact, exactly what it is:

{
  "version": 1,
  "artifacts": {
    "crsp_monthly_clean": {
      "kind": "clean",
      "step": "crsp_monthly_clean",
      "step_version": 1,
      "recipe_hash": "sha256:9f3c…",
      "content_digest": "sha256:1a2b…",
      "inputs": ["crsp_msf@2024-12", "crsp_msenames@2024-12"],
      "rows": 4193021,
      "built_at": "2026-07-05T…"
    },
    "crsp_msf@2024-12": {
      "kind": "raw", "source": "crsp_msf", "vintage": "2024-12",
      "content_digest": "sha256:…", "pit_status": "vintage"
    }
  }
}

  • recipe_hash pins the transform; content_digest pins the actual output bytes (so a corrupted or partial build is detectable independently of the recipe).

  • data_vintage derivation (the contract with numeraire). The data_vintage string a numeraire result carries for a table T is derived directly from T’s lock entry: data_vintage = f"{name}@{recipe_hash[:12]}" (e.g. crsp_monthly_clean@9f3c1d2e4a5b). Given a result’s data_vintage, the lock entry — and thus the full recipe + raw vintages — is recoverable by name and hash prefix. No provenance is stored in two places that could disagree.

WRDS goldens ride the WRDS-CRED tier#

The reproductions that need CRSP/Compustat (W3-2 GKX-lite, W3-3 ALZ empirical, W3-4 HXZ) register their targets as numeraire.golden.GoldenCase(tier=WRDS_CRED, available=<creds+cache present>). That mechanism (built in F3) makes them skip in CI and run verbatim where the user’s WRDS credentials and raw cache are present — one code path, no forked assertions, no licensed bytes in the repo. Live WRDS tests carry @pytest.mark.wrds and are deselected by default.

Boundaries restated (red lines)#

  • WRDS credentials go through env / the tidyfinance WRDS config — never committed, never in code.

  • The raw cache is user-configurable and outside the repo by default.

  • No licensed data (CRSP/Compustat/JKP returns) in any tracked file or git history — ever.