Docs

Recommendation system — north star & staging

Closed causal loop for notification optimization: LLM as policy, causal inference engine as reward function, Bayesian optimization as the loop connecting them, OPE as the safety filter.

Structurally this is RLHF, except the reward model is trained on measured incremental lift instead of human preference labels — a stronger ground truth than almost anything else in the LLM ecosystem.

The final architecture is data-hungry. Most of it is wrong to build early. Build by stage, keyed to what data you actually have.

Ideal architecture (layers)

1. Core abstraction: contextual bandit over an infinite, structured action space

Classical bandits assume a fixed arm set. Here an LLM can generate effectively infinite variants, so the action space is the manifold of possible notifications, not a discrete menu.

1. Embed every variant into a latent space where distance correlates with treatment-effect similarity.
2. Fit a surrogate reward model mapping embedding → predicted incremental lift, with calibrated uncertainty. The label must be the causal engine’s lift estimate, not raw open rate.
3. Acquisition function (Thompson sampling, UCB, or expected improvement) decides which candidates deserve experimental budget.

2. Generator: LLM as amortized proposal distribution

• OPRO / TextGrad-style optimization — feed top-k and bottom-k variants with measured lifts; ask for candidates that push in the winning direction.
• Attribute-conditioned generation — decompose variants into interpretable factors; run the bandit over factor space; let the LLM handle surface realization.

3. Evaluation cascade (OPE pays off here)

Never send a raw LLM idea straight to users. Three gates:

1. Reward model filter — score candidates offline; discard predicted losers.
2. Off-policy evaluation — doubly-robust estimators on logged data before live traffic.
3. Live causal test — survivors get small-traffic experiments via adaptive allocation with a holdout.

4. Full RL framing (maximal theory)

Notifications aren’t one-shot — sends affect future engagement. Ideal formulation: MDP where state = user history/embedding, action = (send/don’t send, variant, time), reward = long-run incremental engagement.

• Timing via temporal point processes (Hawkes) rather than fixed slots.
• In practice: conservative offline RL — online RL on real users is expensive and risky.
• Heterogeneity: estimate CATE per segment; policy becomes personalized assignment.

5. Agentic wrapper

That accumulated causal knowledge base is the real moat — it makes generation improve rather than churn.

Staging (build what the data justifies)

Advance when confidence intervals are tight enough that the bottleneck is idea quality, not measurement precision.

Stage 0 — No data, no money: heuristics + LLM priors

Don’t build ML. Use the LLM as a zero-shot generator with a structured prompt and a fixed attribute schema (urgency, personalization depth, length, framing type, CTA style).

Only engineering that matters: clean logging — variant_id, attributes, user_id, send time, outcome. The knowledge base starts here.

Stage 1 — Hundreds–thousands of sends: A/B over attributes

Run randomized tests that answer attribute-level questions, not variant-level ones. Factorial / factor-level contrasts; LLM fills cells of the design you specify.

Stage 2 — Tens of thousands of sends: verbal optimization loop

• Feed the LLM attribute-level coefficients and top/bottom variants each cycle (OPRO).
• Start Thompson sampling for allocation.
• Keep a 10–20% fixed-randomization holdout so lift estimates stay unbiased.

Stage 3 — ~100k+ sends: surrogate reward model + light heterogeneity

• GBT or ridge on attributes + embeddings → lift, to filter candidates before they cost traffic.
• Split effects by coarse segments (new / active / lapsed).
• Doubly-robust OPE becomes worth implementing.

Stage 4 — Millions of sends: full system

• Bayesian optimization over embedding space
• Sequential testing with always-valid inference
• Offline RL for send-timing and fatigue
• Multi-agent generate / critique / design / synthesize pipeline

Cross-cutting principles

1. Knowledge base from day zero — every causal finding compounds into the generator’s priors.
2. Label = incremental lift, never raw open rate, once you have a causal estimate.
3. Protect randomization (holdout) whenever allocation becomes adaptive.
4. Attribute schema in logging and generation — the bridge from heuristics to models.
5. Advance on CI tightness, not calendar or desire for sophistication.

Where we are today (implementation)

Stage 0 checklist (implemented)

• Fixed attribute schema on every generated variant — pkg/attributes
• Structured generate prompt via POST …/experiments/generate
• Attributes stamped into creative.custom / ClickHouse custom_json on dispatch
• Experiment metrics spreadsheet on the dashboard
• Causal knowledge base the generator reads: data/knowledge/learnings.md

Stack details: Architecture. Inference staging: Inference engine.