I’m comparing OpenAI and DeepSeek models for a product that needs to classify short support messages in several languages. A quick test with translated examples seems too easy: the real messages include slang, typos, and code-switching.

How do you build a fair multilingual evaluation set for this kind of comparison? I’d like to separate language fluency from classification accuracy, and I’m unsure how many native-speaker reviews are enough to trust the results. If you have a benchmark setup that caught a difference a simple English-heavy test missed, I’d be glad to hear about it.