Our support messages often mix English with Hindi written in Latin characters. Customers also use inconsistent spellings, abbreviations, and English product names inside otherwise Hindi sentences.

I want to evaluate a specific DeepSeek model for drafting replies. A benchmark made from clean English-to-Hindi translations doesn’t seem representative of what it would actually receive.

The awkward cases matter most

• A customer switches languages halfway through explaining the problem.
• A sentence includes a negation that changes the requested action.
• An order ID or product name resembles an ordinary word.
• The customer writes in Latin characters but does not explicitly state a preferred reply language.

How would you build a small code-switching evaluation that measures whether the model understood the request, rather than just whether its reply sounds fluent?

I can have bilingual reviewers assess anonymized examples. I’d like to separate intent recognition, preservation of names and numbers, and reply-language choice instead of rolling everything into one quality score. What other failure categories would you include, and would you test model versions separately even when their English results look similar?