Docs

Inference engine — north star & staging

Causal reward function for the notification loop: measure incremental lift honestly, decide when to stop or ship, feed the AI generator calibrated effects — not raw open-rate leaders.

What exists today is a solid stage 1–2 engine. Wilson CIs, Holm correction, and a decision policy that refuses to crown raw open-rate winners put it ahead of most production systems. The staged path forward is mostly about fixing three quiet statistical leaks already present, then adding capability layers in order of data requirements.

The final stack is data-hungry. Most of it is wrong to build early. Build by stage, keyed to what data you actually have.

Ideal architecture (layers)

1. Core abstraction: always-valid inference at the user unit

Randomization unit is the user (sticky hash). Analysis stays at that unit. Every assignment logs the propensity in effect at assignment time. That single logging discipline — assignment, propensity, timestamp, variant attributes — is what makes IPW, OPE, and meta-analysis possible later without re-instrumenting.

2. Honest estimation under adaptive allocation

Thompson-tilted traffic violates the fixed-allocation assumption behind two-proportion z-tests. Arms that got lucky early receive more traffic; naive estimates inherit that selection.

Two fixes (either works; both reuse existing machinery):

• Fixed-split randomized holdout — carve 10–20% of experiment traffic that Thompson never touches; run primary inference there.
• Inverse-propensity weighting (IPW) — store the allocation weight active at each user’s assignment; reweight outcomes.

3. Sequential stopping + final analysis

Peeking on fixed-horizon tests inflates α badly — with daily dashboard checks, real false-positive rates run 2–5× nominal. Stopping labels should come from always-valid inference (mSPRT or confidence sequences / time-uniform bounds for Bernoulli outcomes). Keep the fixed-horizon z-test as an optional “final” analysis; declare_winner must be legitimate at any peek time.

4. Decision policy as expected-loss gate

Decide() today falls back to “ship + rollback flag” when volume is low. The principled version is Bayesian decision theory: compute P(lift > 0) and expected loss of shipping vs waiting, and ship when expected regret of waiting exceeds expected loss of a wrong ship.

Guardrails (unsubscribe / uninstall / notification-disable) are first-class: non-inferiority tests, hard gate in Decide — not just win on opens.

5. Bridge to the recommendation layer

Inference is the reward function the generator reads. Winner lifts must be shrunk (empirical Bayes) before they enter InferenceHint — otherwise inflated estimates corrupt the verbal optimization loop. At scale, doubly-robust OPE scores candidate policies offline before live traffic.

Staging (build what the data justifies)

Trigger to advance stages is statistical, not calendar: move up when current confidence intervals are consistently tight enough that the bottleneck is decision quality or heterogeneity, not measurement noise.

Stage A (now) — Fix the leaks in the current system

1. Adaptive allocation + z-tests is a biased combination. Analyze only a fixed-split holdout slice or apply IPW using assignment-time weights.
2. Peeking on fixed-horizon tests inflates α. Replace the z-test’s stopping role with always-valid inference. Fixed-horizon test can remain as final analysis.
3. Winner’s curse. Fit an empirical-Bayes prior over arm effects and report the posterior mean for the declared winner.

Priority if picking three: sequential inference for stopping → holdout-or-IPW for adaptive allocation → winner shrinkage.

Stage B — Thousands of sends per experiment

• CUPED → CUPAC: residualize against a small ML model predicting outcome from pre-experiment covariates.
• Guardrail metrics as first-class citizens with non-inferiority tests.
• Compliance gap: ITT on assignment; delivered-conditional via IV when delivery failures matter.

Stage C — Tens of thousands+, concurrent experiments

• CATE via coarse segments, then causal forests / DR-learner.
• Expected-loss decisions for the low-volume path.
• Alpha management across experiments (FDR or hierarchical Bayesian pooling).

Stage D — Scale: ideal end state

• Doubly-robust OPE as a service
• Long-run and dynamic effects (novelty decay, fatigue)
• Interference diagnostics where relevant
• Hierarchical meta-analysis for priors and structured generator input

Cross-cutting principles

1. Log propensity at assignment time — enables IPW, OPE, and meta-analysis without re-instrumenting.
2. Analyze at the randomization unit (user sticky hash).
3. Stopping = sequential / always-valid; fixed-horizon tests are optional finals.
4. Protect randomization whenever allocation is adaptive.
5. Feed the AI shrunk, significant lifts — never raw open-rate leaders.
6. Advance on CI tightness and leak severity, not calendar.

Where we are today (implementation)

Baseline checklist (implemented)

• Two-proportion z-test + Wilson CIs; Holm multi-arm correction — pkg/inference
• Decide() policy: ab_test, feature_flag, ship, causal_holdout, adaptive
• Holdout comparison (+ optional CUPED)
• Thompson sampling when allocation: adaptive; weights persist to Redis
• Dashboard + AI generate consume inference (not raw open-rate winners)

Stage A checklist (implemented)

• Always-valid stopping via Bernoulli difference confidence sequences
• Adaptive experiments: default 15% fixed-split holdout; IPW fallback from assignment-time propensity
• Propensity + inference_holdout stamped at assignment
• Empirical-Bayes shrinkage on lifts fed to InferenceHint / AI generate

Stack details: Architecture. Recommendation loop: Recommendation system.