# Data zones: raw → clean → view (design)

> Design note for the WRDS-scale data lifecycle in `numeraire-dataset`. Written before the
> implementation (W3-1). The three-zone split makes preprocessing **reproducible and pinnable**:
> a numeraire result's `data_vintage` string must trace back, unambiguously, to exactly the bytes
> and the exact transform recipe that produced its inputs.

## Why three zones

Preprocessing is part of the method. A Sharpe or an alpha is only reproducible if the *cleaning* —
delisting adjustments, share-code filters, CCM link windows, the accounting-to-return lag — is
pinned as tightly as the model. So the pipeline is split into three zones with hard boundaries:

```
raw/     immutable download cache, keyed by (source, vintage)      — bytes as pulled, never edited
clean/   deterministic transforms of raw → tidy PIT tables         — a recipe hash pins the transform
view/    numeraire TimeSeriesView / CrossSectionView builders      — lazy numeraire import, no new state
```

Arrows point one way: `view` reads `clean`, `clean` reads `raw`, `raw` reads the outside world.
Nothing downstream mutates an upstream zone.

### raw — immutable, pull-stamped

- One directory per `(source, vintage)`, e.g. `raw/crsp_msf/2024-12/`. `vintage` is the pull's
  identity: a WRDS query date, a FRED-MD month, a Ken French release. Content is the bytes exactly
  as pulled (parquet/csv), **never edited in place** — a re-pull with different content is a new
  vintage directory, not an overwrite.
- Each pull writes a sibling `_meta.json`: `{source, vintage, pulled_at, query_hash, pit_status,
  row_count, content_digest}`. `pit_status` records whether the pull is itself point-in-time
  (a dated vintage) or a latest-snapshot convenience pull (revised, not PIT) — so a downstream
  golden can refuse a non-PIT source.
- **The cache directory is user-configurable and defaults OUTSIDE the repo** (`$NUMERAIRE_DATA` →
  platform cache dir, per `paths.data_home`). Licensed data (CRSP/Compustat) lives only here, on
  the user's machine behind their own WRDS credentials — **never in any git history**.

### clean — deterministic, recipe-hashed

- A `clean` table is produced by one or more **steps**: pure `frame(s) → frame` functions with
  **all parameters explicitly declared** (no hidden globals, no ambient config). A step is
  deterministic: same inputs + same params → same output, bit-for-bit.
- Every step's identity is `(name, version, params)`. Its **recipe hash** is the SHA-256 of the
  canonical JSON of `{name, version, params, inputs: [<recipe hash of each input>]}`. The hash
  therefore **chains**: a clean table's recipe hash transitively pins every upstream transform and
  every raw vintage that fed it. Bump `version` when a step's logic changes in a way that should
  invalidate downstream caches.
- Steps register in an **open registry** (the same pattern as numeraire's evaluator registry —
  `register_step` / `get_step` / `available_steps`, a module-global dict, `KeyError` on a duplicate
  name unless `overwrite`). No second registration style is invented. A `@step(...)` decorator is
  sugar over `register_step`.

### view — numeraire builders, lazy

- `view` builders turn a `clean` table into a numeraire `TimeSeriesView` / `CrossSectionView`.
  They import `numeraire` **lazily** (as `sources.to_timeseries_view` already does), so the clean +
  raw zones stay numeraire-free and installable without it. `view` adds no persisted state.

## `data.lock.json`

A single lock file records, for each built `clean` (and pinned `raw`) artifact, exactly what it is:

```json
{
  "version": 1,
  "artifacts": {
    "crsp_monthly_clean": {
      "kind": "clean",
      "step": "crsp_monthly_clean",
      "step_version": 1,
      "recipe_hash": "sha256:9f3c…",
      "content_digest": "sha256:1a2b…",
      "inputs": ["crsp_msf@2024-12", "crsp_msenames@2024-12"],
      "rows": 4193021,
      "built_at": "2026-07-05T…"
    },
    "crsp_msf@2024-12": {
      "kind": "raw", "source": "crsp_msf", "vintage": "2024-12",
      "content_digest": "sha256:…", "pit_status": "vintage"
    }
  }
}
```

- `recipe_hash` pins the transform; `content_digest` pins the actual output bytes (so a corrupted
  or partial build is detectable independently of the recipe).
- **`data_vintage` derivation (the contract with numeraire).** The `data_vintage` string a numeraire
  result carries for a table `T` is derived *directly* from `T`'s lock entry:
  `data_vintage = f"{name}@{recipe_hash[:12]}"` (e.g. `crsp_monthly_clean@9f3c1d2e4a5b`). Given a
  result's `data_vintage`, the lock entry — and thus the full recipe + raw vintages — is recoverable
  by name and hash prefix. No provenance is stored in two places that could disagree.

## WRDS goldens ride the `WRDS-CRED` tier

The reproductions that need CRSP/Compustat (W3-2 GKX-lite, W3-3 ALZ empirical, W3-4 HXZ) register
their targets as `numeraire.golden.GoldenCase(tier=WRDS_CRED, available=<creds+cache present>)`.
That mechanism (built in F3) makes them **skip in CI** and run verbatim where the user's WRDS
credentials and raw cache are present — one code path, no forked assertions, no licensed bytes in
the repo. Live WRDS tests carry `@pytest.mark.wrds` and are deselected by default.

## Boundaries restated (red lines)

- WRDS credentials go through env / the tidyfinance WRDS config — never committed, never in code.
- The raw cache is user-configurable and outside the repo by default.
- No licensed data (CRSP/Compustat/JKP returns) in any tracked file or git history — ever.