Docs

Notification observability architecture

Two-sided event collection for third-party push notifications: backend dispatched events plus client delivered / opened / dismissed, correlated by notification_id.

Patterns

Pattern A — Push Proxy (primary)

Customer Backend → POST /v1/send (push-proxy:8080)
  → generate notification_id
  → inject into payload data
  → real APNs when Connect credentials are ready, else mock APNs/FCM
  → Kafka notification.events (dispatched + creative)

Pattern B — Dual ingestion (decoupled)

Customer Backend → own push vendor
                 → POST /v1/events (ingest-server:8081)  // dispatched
Mobile SDK      → POST /v1/client/events (ingest-client:8082)  // delivered/opened/dismissed

Pipeline

HTTP APIs → Redpanda (Kafka) → stream-processor → ClickHouse + Redis
                                              ↘ query-api / dashboard

Component	Port	Role
push-proxy	8080	Pattern A dispatch
ingest-server	8081	Pattern B server events
ingest-client	8082	SDK client events
query-api	8083	Metrics + creative analytics + UI
mock-push-provider	8090	Local FCM stub + APNs fallback when Connect is empty
Redpanda	19092	Kafka-compatible bus
ClickHouse	8123 / 9000	Historical analytics
Redis	6379	Live counters + dispatch context + experiment configs
Grafana	3000	Optional dashboards

Creative v1

On dispatched, the creative snapshot is stored in notification_creatives:

title, body, image_url, deep_link, locale, custom (JSON)
experiment_id, variant_id when sent via an experiment
Joined to funnel events via notification_id

A/B experimentation

Experiments live in Redis (experiment:{app}:{id}), managed via push-proxy and query-api:

Endpoint	Role
`POST /v1/apps/{app}/experiments`	Create (draft or running) — push-proxy or query-api
`GET/PATCH/DELETE …/experiments/{id}`	Read / update status or variants / delete (push-proxy)
`POST …/experiments/{id}/assign`	Preview sticky assignment
`POST …/experiments/generate`	OpenAI-suggested variants (query-api; needs `OPENAI_API_KEY`)
`POST /v1/send` with `experiment_id`	Assign variant, stamp creative, dispatch

Assignment is sticky: SHA-256(experiment_id:recipient_id) % 10000 mapped onto variant weights. Optional variant_id on send forces a specific arm.

Events and creatives carry experiment_id / variant_id. Query API exposes:

GET /v1/apps/{app}/cuts — funnel cuts by template, channel, device_os, locale, campaign, experiment, variant
GET /v1/apps/{app}/experiments/{id}/metrics — per-variant funnel rates
POST /v1/apps/{app}/experiments/generate — AI next-round creatives (optional source_experiment_id + metrics)

AI generate → create → send loop

Run an experiment and review variant metrics on the dashboard.
Generate variants (brief and/or selected experiment as source) via query-api → OpenAI.
Preview, then Create & run — persists a running experiment in Redis.
Dispatch with push-proxy POST /v1/send including experiment_id.

Set OPENAI_API_KEY (and optional OPENAI_MODEL, default gpt-4o-mini) on query-api. Without a key, generate returns 503.

Correlation

Proxy (or customer) assigns globally unique notification_id.
ID (and experiment/variant when present) travel in push data to the device.
SDK echoes the same ID on delivered/opened/dismissed.
Stream processor computes latency_ms using Redis notif:{id}.dispatch_ts and backfills experiment fields onto client events.

Auth (local)

Header X-API-Key: demo-key mapped to app_gh8932 via API_KEYS=demo-key:app_gh8932.

ClickHouse local credentials: default / notifications.

Query-api generate/create are open locally for the dashboard demo; gate with API keys in production.

Local run

export OPENAI_API_KEY=sk-...   # optional; enables AI variant generation
docker compose up --build -d
./scripts/seed.sh
open http://localhost:8083/

Also see Inference engine and Recommendation system.