We’re preparing to launch a support assistant using an OpenAI model, and I’m trying to work out what a realistic evaluation set should look like. Our real tickets contain personal details, but synthetic examples often miss the awkward edge cases agents actually see.

How are people balancing privacy with coverage? I’m especially interested in whether a small evaluation set can still catch failures like misread intent, incorrect escalation, or confidently wrong policy answers. What do you measure beyond whether the answer looks plausible?