ab-testing

Files (2)

• README.md
• SKILL.md

SKILL.md (first lines)

# A/B Testing

## Rules

- Hash-based assignment: deterministic variant selection using `hash(userId + experimentId) % 100` — no randomness drift between sessions
- Variant config: store experiments in a config object `{ id, name, variants: [{ id, weight }], status }` — never hardcode variants
- Split traffic server-side: assign in middleware or API route, set cookie for consistency — client-side splits cause flicker
- Statistical significance: require 95% confidence (p < 0.05) before declaring a winner — use z-test for proportions
- Sample size: calculate minimum sample before starting — `n = (Z^2 * p * (1-p)) / E^2` where E is minimum detectable effect
- One metric per experiment: define a primary metric upfront (conversion rate, click-through, revenue) — secondary metrics are exploratory only
- Mutual exclusion: users in one experiment should not overlap with conflicting experiments — use experiment layers
- Feature flags: wrap experiment code in feature flag checks — clean up losing variants after experiment ends
- Duration: run for at least 1-2 full business cycles (7-14 days minimum) to account for day-of-week effects
- Logging: track `experiment_id`, `variant_id`, `user_id`, `timestamp` for every impression and conversion event

## Avoid

- Peeking at results before sample size is reached — inflates false positive rate
- Changing experiment parameters mid-run — invalidates statistical analysis
- Testing too many variants at once — increases required sample size exponentially
- Client-side variant assignment without cookies — causes flickering and inconsistent experiences

From YepAPI/skills, MIT.

More from YepAPI/skills