I’m considering changing the OpenAI model behind an internal writing assistant. It turns rough project notes into updates for different audiences, so there usually isn’t one reference answer that every good response should match.

One model might produce a shorter update; another might explain the risks more clearly. Both can be useful. What I don’t want is to approve an upgrade because the wording sounds better while it quietly drops an unresolved dependency.

What would make a fair comparison?

• Preserving decisions, deadlines, and uncertainty from the notes.
• Following the requested audience and length.
• Not inventing progress or commitments.
• Producing something people can use with minimal editing.

I’m leaning toward blind pairwise evaluation, with reviewers choosing between outputs without seeing the model names. But should factual omissions be scored separately rather than folded into an overall preference?

For anyone doing OpenAI model upgrade testing, how do you separate genuine improvements from a preference for a particular writing style? I’d also like to know how you handle cases where reviewers reasonably disagree.