Data zones: raw → clean → view (design)#

Design contract for the WRDS-scale data lifecycle in numeraire-dataset. The three-zone split makes preprocessing reproducible and pinnable: a numeraire result’s data_vintage string must trace back, unambiguously, to the exact rows, schema, query, and transform recipe that produced its inputs.

Why three zones#

Preprocessing is part of the method. A Sharpe or an alpha is only reproducible if the cleaning — delisting adjustments, share-code filters, CCM link windows, the accounting-to-return lag — is pinned as tightly as the model. So the pipeline is split into three zones with hard boundaries:

raw/     immutable download cache, keyed by (source, vintage)      — pulled frame, never edited
clean/   deterministic transforms of raw → tidy PIT tables         — a recipe hash pins the transform
view/    numeraire TimeSeriesView / CrossSectionView builders      — lazy numeraire import, no new state

Arrows point one way: view reads clean, clean reads raw, raw reads the outside world. Nothing downstream mutates an upstream zone.

raw — immutable, pull-stamped#

  • One directory per (source, vintage), e.g. raw/crspm_msf/2024-12/. vintage is the pull’s identity: a WRDS query date, a FRED-MD month, a Ken French release. The returned rows and schema are serialized to parquet/csv and never edited in place — a re-pull with different content is a new vintage directory, not an overwrite.

  • Each pull writes a sibling _meta.json: {source, vintage, pulled_at, query_hash, pit_status, row_count, content_digest, content_digest_version}. pit_status records whether the pull is itself point-in-time (a dated vintage) or a latest-snapshot convenience pull (revised, not PIT) — so a downstream reference check can refuse a non-PIT source. Digest contract version 2 covers values plus extension dtypes, categorical levels/order, timezone, object-scalar type and missing sentinel identity, and column/index metadata. Source, vintage, exact query hash, PIT status, content digest, and its contract version form one canonical raw-input identity; all six are chained into each downstream clean recipe hash. Identical bytes relabeled from snapshot to vintage therefore cannot retain the same clean data_vintage.

  • A cache hit is accepted only when the sidecar query_hash matches the requested SQL. If a standard query gains columns while the human-readable source/vintage label stays the same, the reader fails closed and requires a new vintage; it never silently serves the old schema under the new recipe. The hash covers exact SQL text: ordinary whitespace is not folded because whitespace inside a string literal can change selected rows. The same identity/row/schema/value validation path serves read_raw, pull_raw, pull_standard, and clean builds, so a lower-level read cannot bypass the lock contract.

  • Parquet and _meta.json are written under the dedicated data-home staging namespace, then the complete directory is atomically renamed into place. A complete raw (source, vintage) directory is immutable, so refresh=True cannot overwrite it. A legacy/incomplete directory containing only one of the two known artifacts was never committed and can be safely rebuilt; unexpected files fail closed for manual inspection. An OS-released per-vintage writer lock makes concurrent first pulls execute the licensed query once and adopt one validated result; after a killed process, the next lock owner removes only recognized abandoned staging artifacts. The public low-level write_meta helper is create-only—it cannot relabel an existing cache. Locks and staging use content-hashed names in dedicated data-home namespaces. Valid labels are single path components; @ and # are reserved for lock/chaining identities, so labels cannot collide with another vintage or broaden cleanup scope. Every internal namespace component is checked before and after creation; symlinked staging/lock paths fail closed, and final parquet/metadata artifacts must be regular non-symlink files, so reads and cleanup cannot traverse outside the configured data home.

  • The cache directory is user-configurable and defaults OUTSIDE the repo ($NUMERAIRE_DATA → platform cache dir, per paths.data_home). Licensed data (CRSP/Compustat) lives only here, on the user’s machine behind their own WRDS credentials — never in any git history.

Partitioned daily equity collections#

Daily CRSP security history is too large for the standard monthly pull_standard/load_clean surface. zones.wrds_equity reuses the same immutable pull_raw cache for each stable mod(permno, partitions) bucket (64 by default), then binds all buckets plus CRSP DSI vwretd and ff.factors_daily.rf into a canonical manifest under collections/. A single immutable raw marker for the same vintage records max(crspm.msf_v2.mthcaldt). Immediately before any uncached bucket and again after all pulls, that cached value is compared with a fresh one-cell WRDS query. A mismatch fails before manifest publication and requires a new vintage, preventing a resumed long pull from combining two CRSP releases. The manifest is published only when every independent raw input is complete. It pins the marker plus exact query and content digests, rows/date bounds, SIZ-or-CIZ convention, decimal unit contracts, each registered clean-step name/version and explicit override, and a derived data_vintage; it contains no SQL text, credentials, licensed rows, or local path.

The collection deliberately has no concatenate method. Iteration revalidates and loads one parquet at a time, so peak client memory is bounded by one bucket. SIZ (legacy crspm.dsf common shares) is the primary paper-era convention. CIZ (dsf_v2 with CRSP’s mapped common-equity definition) is a separate sensitivity whose query/cache/manifest identity cannot be mixed with SIZ.

The market/RF cleaner requires identical calendars and decimal simple-return units. WRDS daily RF currently starts in July 1926, so a CRSP range beginning in January 1926 fails on the uncovered sessions; missing early RF is never replaced by zero or a forward fill. The effective BAB daily start is therefore 1926-07-01, which still clears the paper’s 750-session warm-up before April 1929.

Monthly security targets use a separate SIZ-only collection. Each bucket mechanically contains MSF, date-overlapping MSENAMES, then MSEDELIST, followed after all buckets by one monthly FF RF input. The same leading CRSP release marker and before/after live gate protect this collection. The three security inputs are queried independently—never through an MSF inner join—so a delist-only terminal month survives. The standard clean iterator applies no exchange screen and no unreported performance-delist imputation, while retaining source_ret, dlret, adjusted ret, month-end decimal RF, and excess_ret. WRDS labels monthly RF at the first of its return month; a strict cleaner verifies decimal units and uniqueness before moving that label to month-end. The paper-facing iterator is intentionally fixed. Sensitivity transforms start from the raw partition iterator and use the open step registry, producing a separate recipe identity rather than reusing the collection’s data_vintage for different rows.

clean — deterministic, recipe-hashed#

  • A clean table is produced by one or more steps: pure frame(s) → frame functions with all parameters explicitly declared (no hidden globals, no ambient config). A step is deterministic: same inputs + same params → same output, bit-for-bit.

  • Every step’s identity is (name, version, params). Recipe-hash contract version 2 hashes the canonical JSON of {recipe_hash_version, name, version, params, inputs: {role: input_hash}}. Binding each declared input role prevents a non-commutative step from colliding when two inputs are exchanged. An explicitly supplied input hash is used when available; otherwise run_step hashes that input frame’s values and semantic schema, so the default and partial-mapping APIs remain data-sensitive. Parameter values are recursively type-tagged and frozen at call entry; stateful values without a complete canonical form fail closed. The public recipe_hash helper therefore accepts {input_role: hash}, not the legacy bare hash list. The hash chains: a clean table transitively pins every upstream transform and raw vintage that fed it. Bump a step’s version when its own logic changes. Unknown frame or input-hash roles are rejected, so a typo in a pinned role cannot silently fall back to hashing only the frame and lose source/query/PIT provenance; intentionally omitted hashes still use the documented content-digest fallback.

  • Steps register in an open registry (the same pattern as numeraire’s evaluator registry — register_step / get_step / available_steps, a module-global dict, KeyError on a duplicate name unless overwrite). Every transform parameter must be explicitly named; variadic **kwargs is rejected so keyword insertion order cannot become uncommitted executable state. No second registration style is invented. A @step(...) decorator is sugar over register_step.

view — numeraire builders, lazy#

  • view builders turn a clean table into a numeraire TimeSeriesView / CrossSectionView. They import numeraire lazily (as sources.to_timeseries_view already does), so the clean + raw zones stay numeraire-free and installable without it. view adds no persisted state.

data.lock.json#

A single lock file records, for each built clean (and pinned raw) artifact, exactly what it is:

{
  "version": 2,
  "artifacts": {
    "crsp_monthly_clean": {
      "kind": "clean",
      "step": "crsp_monthly_clean",
      "step_version": 3,
      "recipe_hash": "sha256:9f3c…",
      "recipe_hash_version": 2,
      "params": {},
      "content_digest": "sha256:1a2b…",
      "content_digest_version": 2,
      "inputs": {
        "msf": {"artifact": "crspm_msf@2024-12", "hash": "raw-input-v1:{…}"},
        "msenames": {"artifact": "crspm_msenames@2024-12", "hash": "raw-input-v1:{…}"},
        "msedelist": {"artifact": "crspm_msedelist@2024-12", "hash": "raw-input-v1:{…}"}
      },
      "rows": 4193021,
      "data_vintage": "crsp_monthly_clean@9f3c1d2e4a5b"
    },
    "crspm_msf@2024-12": {
      "kind": "raw", "source": "crspm_msf", "vintage": "2024-12",
      "content_digest": "sha256:…", "content_digest_version": 2,
      "query_hash": "sha256:…", "pit_status": "vintage"
    }
  }
}

  • Lock format version 2 records every type-tagged canonical explicit parameter override and, for each input role, its artifact label and exact chained hash. Defaults are intentionally interpreted by the recorded step name/version in the pinned package code; they are not duplicated in the lock where they could drift. The entry is therefore auditable rather than merely an opaque hash. recipe_hash pins that transform plus role-bound input mapping; content_digest pins ordered values and semantic schema, including column/index metadata, extension dtype, categorical order, timezone, and object-scalar type/missing-sentinel identity (so a corrupted, partial, or type-reinterpreted build is detectable independently of the recipe). Lock insertion recomputes that digest and rejects a Built.frame or digest field mutated after run_step; the row count and digest therefore describe the same output snapshot. A missing content_digest_version denotes the legacy value-only v1 contract; it must not be interpreted as a v2 digest. Lock format version 1 used the legacy untyped parameter/input-list recipe and is rejected rather than silently reinterpreted.

  • data_vintage derivation (the contract with numeraire). The data_vintage string a numeraire result carries for a table T is derived directly from T’s lock entry: data_vintage = f"{name}@{recipe_hash.split(':', 1)[-1][:12]}" (for example, crsp_monthly_clean@9f3c1d2e4a5b). Given a result’s data_vintage, the lock entry recovers the registered step/version, explicit overrides, and role-bound raw vintages. Together with the pinned package version, these determine the full executable recipe. No parameter override or input binding is hidden behind the digest alone.

Credentialed reference checks#

Reproductions that need CRSP or Compustat register their targets with numeraire.reference.ReferenceResult(tier=CREDENTIALED, available=<credentials-and-cache-present>). They skip in public CI and run verbatim where the user’s WRDS credentials and raw cache are present — one code path, no forked assertions, and no licensed bytes in the repo. Live WRDS tests carry @pytest.mark.wrds and are deselected by default.

Boundaries restated (red lines)#

  • WRDS credentials come from the environment by default, or explicit in-process connector arguments; they are never persisted by this package, embedded in generated SQL, or committed.

  • The raw cache is user-configurable and outside the repo by default.

  • No licensed data (CRSP/Compustat/JKP returns) in any tracked file or git history — ever.