I’m building a customer-support assistant with the OpenAI API. Answer accuracy is fairly easy to check against our help articles, but I’m less confident about its decisions to keep helping versus transfer the conversation to a person.

For example, “Where is my refund?” could be a routine status question, a missing payment that needs investigation, or a complaint after three unsuccessful support contacts. A keyword rule seems too blunt.

Cases I want the evaluation to cover

• The customer explicitly requests a human.
• The assistant has already suggested a step that failed.
• The request needs account access or authority the assistant does not have.
• The customer sounds frustrated, but the question is still straightforward.

How would you build an evaluation set for customer support escalation decisions? Would you score unnecessary transfers separately from missed escalations, and how would you handle examples where two support agents disagree about the right next step?

I’m interested in testing the decision across a whole conversation, not just classifying its last message.