Our support team sometimes gets messages that switch between English and another language mid-sentence, including product names and technical terms that customers leave untranslated. I’m considering DeepSeek for first-pass classification and drafting, but standard translation tests don’t seem like a good fit for this kind of input.

What would you include in an evaluation set for code-switching conversations? I’m trying to figure out how to judge whether the model preserves the customer’s meaning and handles mixed-language terminology consistently, rather than just producing fluent-looking text.