User guide#

The package has two layers: sources (tidyfinance-primary loaders for standard data) and builders (self-built ETL for what tidyfinance does not cover). Reference reproductions keep their own pinned fixtures; the loaders here are for convenience, exploration, and building those fixtures.

Sources#

The source loaders return tidy pandas frames. tidyfinance is the primary backend; portfolio frames carry inspectable source provenance, and the view helpers return their data_vintage explicitly.

from numeraire_dataset import (
    load_ff_factors,
    load_ff_momentum,
    load_ff_portfolios,
    load_goyal_welch,
    load_hxz_q4_factors,
    load_hxz_q_factor_legs,
    load_pastor_stambaugh_historical_beta_portfolios,
    load_pastor_stambaugh_liquidity,
)

ff = load_ff_factors()          # decimal FF3/RF; monthly observations available at month-end
industry = load_ff_portfolios(portfolio_set="industry_10")  # date + ten decimal returns
daily_mom = load_ff_momentum(freq="daily")                  # date + canonical decimal `mom`
gw = load_goyal_welch()         # the Goyal–Welch predictor set
liquidity = load_pastor_stambaugh_liquidity()
liq_deciles = load_pastor_stambaugh_historical_beta_portfolios()
q4 = load_hxz_q4_factors(start_date="1972-01-01", end_date="2012-12-31")
q4_legs = load_hxz_q_factor_legs(start_date="1972-01-01", end_date="2012-12-31")

load_ff_portfolios also supports portfolio_set="size_bm_25" and "momentum" (the Ken French Mom/WML factor) at monthly and daily frequencies. Because Mom is a factor, load_ff_momentum is its clearer primary name; the portfolio-shaped compatibility path delegates to it. Monthly observations are timestamped at month-end, when the return is realized, rather than at the source’s month-start period label.

load_ff_factors uses the same month-end convention. Its inclusive start_date / end_date bounds are applied after that normalization; provenance records both the widened month-start backend query and the exact selected month-end range, so a mid-month boundary cannot silently drop the first month or return a timestamp beyond the requested end.

Daily Mom is one documented exception to the tidyfinance backend: tidyfinance currently returns no observations for that release, so the loader downloads the official Ken French CSV zip directly into memory. It requires the published header/footer, preserves trading-day dates, converts percent returns to decimals, and rejects duplicate dates, source missing sentinels, and non-finite values. No raw file is written. frame_provenance(daily_mom) includes the official HTTPS URL, HTTP source as-of timestamp, content SHA-256, and parsing recipe; frame_data_vintage(daily_mom) returns the non-secret source-vintage prefix of the value returned by load_ff_portfolio_view(portfolio_set="momentum", freq="daily"); the latter appends its complete-case view recipe.

Hou–Xue–Zhang q-factor release#

load_hxz_q4_factors and load_hxz_q_factor_legs are strict, memory-only adapters for the official Global-q release supporting Hou, Xue, and Zhang (2015). Both pin the 2025-02-16 source bytes, validate the complete 1967-01–2024-12 release before an inclusive date slice, convert the source’s percent returns to decimals, and accept redirects only to HTTPS global-q.org. No source CSV is cached or packaged; provenance records the fixed URL/release/SHA, parser and unit recipe, full and selected row counts, and redistributable=false.

The release’s 1967–1971 observations are a backward extension relative to the paper. Use the explicit 1972-01–2012-12 bounds shown above when reproducing the paper sample; omitting bounds deliberately returns the complete current release instead.

The current factor file is a q5 carrier. The original HXZ model is deliberately frozen as HXZ_Q4_FACTOR_COLUMNS = ("mkt_excess", "me", "ia", "roe"): the parser validates R_EG but does not expose it through load_hxz_q4_factors. The output has exactly date, risk_free, mkt_excess, me, ia, roe, with month-end timestamps. The legs output has one row for every monthly (rank_me, rank_ia, rank_roe) cell in the 2x3x3 grid, positive nstocks, and decimal ret_vw / retx_vw. Use ret_vw when rebuilding the published factors; retx_vw is retained so the dividend convention remains auditable.

Pastor–Stambaugh liquidity releases#

The paired loaders use the year-stamped files on the official author data page, download the response once into memory, and write no cache or fixture. Both frames use month-end timestamps. The method reference is Pastor and Stambaugh (2003). Their provenance includes the requested and resolved Booth URL, an allowlist of HTTP metadata, retrieval time, exact response SHA-256, parser/unit recipe, and the requested and selected date slice. data_vintage depends on the content hash and recipe, not on a mutable Last-Modified timestamp. A newer year-stamped URL can be supplied explicitly, but it must remain in the same official Booth author-data directory; the Wharton mirror is never used as a silent fallback.

The aggregate frame has five columns:

  • agg_liq and innov_liq preserve the author’s downloaded numeric scale;

  • liquidity_innovation = innov_liq / 100 is the explicit feature contract used by the Pastor–Stambaugh equations 14–18 reproduction and numeraire-zoo;

  • traded_liq is the decimal return on the author’s historical-beta 10-minus-1 portfolio. Its documented -99 observations before 1968 become missing; no other missing sentinel is accepted.

The portfolio frame contains decimal total returns in decile_1 through decile_10, plus the author-supplied decimal traded_liq. It has no source date column; the parser constructs one row per month from the author header, validates the row count, and checks the supplied spread against decile_10 - decile_1 without replacing it. Equation 14 needs excess test-asset returns, so join risk_free from load_ff_factors() and subtract it from each decile. The long-short spread itself is unchanged by that subtraction.

These releases are full-history, latest/revised snapshots rather than historical point-in-time vintages; a month label must not be interpreted as evidence that the revised value was knowable at that time. The downloadable portfolios are sorted on historical liquidity betas beginning in 1968. The paper’s Table 6 instead uses predicted-beta portfolios beginning in 1966, so the download cannot support a Table 6 exact-reproduction claim. It can support a current-vintage run of the historical-beta design and comparisons with the paper’s Table 8, with vintage drift disclosed.

Daniel–Moskowitz local momentum archive#

load_daniel_moskowitz_momentum(path, contract=...) accepts only a local caller-obtained DM_data_2014_02.tar.gz; it performs no download, discovery, extraction, copy, or cache write. The archive must contain exactly the twelve root-level regular files documented by the authors. DANIEL_MOSKOWITZ_ARCHIVE_MEMBERS exposes the immutable name allowlist. Path traversal, links, devices, duplicate/unexpected names, incomplete member sets, and compressed/uncompressed size or ratio violations fail before parsing. Bounded physical-line, parsed-record, column, aggregate-field, and record-length counts prevent a small text carrier from amplifying into unbounded Python objects. Secure traversal of every source-path component requires POSIX dir_fd and O_NOFOLLOW support; platforms without those primitives fail closed before any archive bytes are read.

There is deliberately no built-in text-layout guess. The public source documentation establishes the filenames and portfolio meaning, but not a machine-readable delimiter, header, and column order. Supply a frozen DanielMoskowitzArchiveContract containing distinct daily and monthly DanielMoskowitzMemberContract values. Each member contract declares:

  • the exact allowlisted member, delimiter and text encoding (whitespace means ASCII whitespace);

  • zero-based non-empty header/data record bounds;

  • the date column, ten ordered decile-return columns, and explicit date format;

  • source units (decimal or percentage points); and

  • optional expected archive/member SHA-256, first/last date, row count, or exact date sequence.

The paired result has daily and monthly frames with exactly date, decile_1, ..., decile_10, wml. Returns are decimal simple returns, wml is decile 10 minus decile 1, monthly dates are month-end and gap-free, daily dates remain the source trading dates, and the two members must cover the same calendar months. An optional alignment tolerance can require daily deciles to compound back to the monthly deciles. frame_provenance records actual archive, member, paired-member, all-member-manifest, and parser-contract hashes without recording the local path. Unless expected hashes were supplied, identity is marked as recorded but incompletely pinned; the loader itself never makes a paper-exact claim.

The archive does not contain the dynamic strategy’s market-state inputs. Source monthly market total returns, daily market excess returns, and risk-free rates separately and retain their own provenance. In particular, do not use excess returns for the 24-month bear signal or total returns for the 126-session market-variance input.

He–Kelly–Manela fixed paper archive#

load_hkm_paper_data(frequency="quarterly" | "monthly", path=None) validates the frozen official He–Kelly–Manela archive before returning the original 1970Q1–2012Q4 test-asset sample. The default route downloads the one official URL into memory; path= instead requires a byte-identical local ZIP and uses POSIX dir_fd/O_NOFOLLOW traversal. Unsupported local-file platforms fail closed. Neither route extracts, caches, copies, executes, or persists a member.

The ZIP must contain exactly its seven documented root-level regular members. Duplicate or unsafe paths, links, encryption, hidden ZIP metadata, unsupported compression, size/ratio violations, and schema/calendar drift are rejected. The three updated factor-only files are validated as part of the envelope but have no public selection route: the paper test-assets API cannot silently expand through their 2018 endpoint or switch to daily data.

HKMPaperData contains:

  • factors: decimal paper factors, with mkt_rf and rf canonically named mkt_excess and risk_free;

  • excess_returns: 124 distinct paper assets in seven classes, each source total/net return minus same-period RF, retaining every source missing value;

  • asset_metadata: asset, class, within-class position, and its redundant source All_* column.

The source All_01–All_124 cells must exactly duplicate the seven named class blocks. They are validated but not returned as another 124 assets. data.complete_case("us_bonds") selects dates where all U.S. bond assets and factors exist; another class’s shorter history has no effect.

All three frames support frame_provenance and record the fixed archive/member hashes, rows, unit conversion, class contract, source mode, and a path-free vintage. The author permits free non-commercial use and supplies the archive as-is, without a general redistribution grant; the paper’s third-party test-asset citations still apply. Provenance therefore says redistributable=false, and package distributions exclude the ZIP, CSV, and Julia carriers.

Frame loaders vs. view helpers#

The split is intentional. Frame loaders (load_ff_factors, load_ff_momentum, load_ff_portfolios, load_goyal_welch, the HXZ loaders, the Pastor–Stambaugh loaders, and the Daniel–Moskowitz and HKM archive loaders) return plain tidy frames and carry no numeraire dependency, so they are usable on their own. HKM groups three frames in a frozen result object but still has no view or numeraire dependency. The view helpers (load_ff_portfolio_view, load_gw_view, to_multiasset_view, to_timeseries_view) add the optional bridge into a numeraire TimeSeriesView plus a data_vintage stamp, importing numeraire lazily (install the [numeraire] extra):

from numeraire_dataset import load_ff_portfolio_view, load_gw_view

assets, assets_vintage = load_ff_portfolio_view(portfolio_set="industry_10")
view, vintage = load_gw_view(start_date="1926-07-01", end_date="2020-12-31")
# view -> feed straight into numeraire's backtest; vintage -> the provenance string

Builders#

Self-built ETL for what tidyfinance does not provide. The flagship is vintage-aware FRED-MD: a real-time macro panel indexed by reference period × vintage × series, with the FRED-MD stationarity transforms (tcodes) applied at build time, per vintage — so revisions are first-class and an asof read is leak-safe.

from numeraire_dataset.builders import fredmd

paths = fredmd.download(vintages=["2025-01", "2025-02", "2025-03"], dest="~/.numeraire_data")
table = fredmd.build_table(paths, transform=True)   # tidy [reference, vintage, series…]

transform=False keeps raw levels. The availability lag stays a read-time parameter in numeraire (not baked into the table), so you can sweep it for robustness. See API reference for the full builder surface (download, download_archive, read_vintage, apply_tcode, build_table, build_from_dir).

Data zones (WRDS-scale)#

For subscription panels (CRSP / Compustat, via your own WRDS account), the package uses a three-zone raw → clean → view lifecycle that pins preprocessing as tightly as the model, so a result’s data_vintage traces back to the exact normalized rows/schema/query and transform recipe. The design is described in Data zones: raw → clean → view (design).

Annual Compustat exposes two intentionally different clean contracts. compustat_annual_clean retains its configurable datadate + report_lag_months heuristic for compatibility. compustat_annual_formation_clean instead preserves the raw revt, cogs, at, gp, fyear, fyr, and sic fields and stamps every fiscal year ending in calendar year t with a June 30, t+1 formation_date. It does not fill missing cost of goods sold, filter on book equity, or construct a method-specific profitability signal.

The CRSP clean table retains stock months and delist-only terminal months. is_terminal, dlret, and dlstcd make the terminal-return convention visible; a delist-only row keeps missing market capitalization instead of inventing a price or share count.

Raw WRDS vintages are immutable only after parquet and _meta.json have been atomically published together. Every cache-reading API validates the stored query, row count, values, and semantic schema. The schema-aware digest distinguishes nullable/extension dtypes, categorical order, and timezone, plus object-scalar types and missing sentinels, even when displayed values or pandas’ default value hashes match. A cache created under an older digest contract must be rebuilt under a new vintage label. Clean recipe hashes bind the raw source/vintage, query hash, semantic digest and digest-contract version, and snapshot/vintage PIT status. Consequently, changing only a raw source’s methodological PIT label still moves the clean data_vintage; consult the matching lock entry to audit the complete identity.