Prove the change on live traffic
Experiments run A/B on the real Shinobi agent, not a sandbox. Arm A is your current behavior, byte for byte untouched. Arm B changes exactly one thing: an extra plain-language rule or a different greeting. Containment and CSAT are measured per arm, and nothing is called a winner without significance.
Arm A untouched, arm B moves one lever
An experiment is a hypothesis with exactly one change behind it. Because arm B differs from arm A by a single lever, a lift means the lever caused it, not a pile of changes you cannot untangle.
Current behavior, untouched
The control is not a reconstruction. Arm A conversations run the agent you ship today, exactly as it is, so the baseline is real by definition.
One extra rule, applied last
Write the change as a plain-language workspace rule. Arm B turns get it after your own rules, so it always has the last word, and arm A never sees it.
A different first message
Or test the opener: arm B conversations start from your candidate greeting while arm A keeps the one you have. Same agent behind both.
Every conversation is dealt once
When a new conversation starts, it is dealt an arm on the spot: a hash of the experiment and the session id, at the split percentage you set. The same session always lands in the same arm, an existing conversation never switches mid-thread, and one experiment runs at a time so every session's assignment stays unambiguous.
A verdict you can take to a meeting
Each arm is scored on the sessions it actually got. Contained means the conversation never needed a human: no assignment, no case filed. CSAT comes with the rating count behind it. The containment difference goes through a two-proportion z-test, and under 20 conversations per arm the verdict says not enough conversations yet. Nothing is called a winner without p < 0.05.
One click makes the winner the config
Adopt arm B appears only on a significant, positive result. Clicking it writes the real thing: the rule joins your workspace rules, or the greeting becomes the greeting, and the experiment stops. No copy step, no drift between what won and what runs. Experiments is the proof stage of the flywheel: Scout finds the question, Training proves the behavior in simulation, Self-Healing ships the fix, and Experiments proves it on live traffic.
Live today: every new conversation is hashed into an arm at session start, at the split you set, and stays there for its whole life.
Live today: a two-proportion z-test on containment, a 20-conversation minimum per arm, and a verdict that stays open until p < 0.05.
Live today: a significant positive result unlocks Adopt arm B, which writes the rule into your workspace rules or makes the greeting the greeting, then stops the experiment.
Live today: one running experiment per workspace. Starting a second is refused with the reason, so every session's arm is always certain.