I’m testing OpenAI’s audio transcription for interview recordings. Most of the transcript reads well, but the mistakes that matter most are people’s names, company names, and short product codes. A smooth-looking sentence can hide a completely wrong name.

I have a small set of recordings with manually checked transcripts. They include remote calls, overlapping speech, and speakers mentioning names that sound like ordinary English words.

What would be a fair test?

• Should I track proper noun accuracy separately from the overall word error rate?
• If the selected endpoint supports contextual hints, should I test with a full glossary or only names relevant to that recording?
• How can I check whether a glossary causes the model to insert an expected name when the speaker actually said something else?

I’m especially interested in how people handle genuinely unclear audio in the reference transcript. I don’t want to penalize reasonable uncertainty, but I also don’t want confident guesses counted as correct. What would you include in an audio transcription evaluation before using this for searchable interview archives?