I’m choosing between quantized builds of the same DeepSeek model for a local support tool. Fluent explanations matter, but the less forgiving requirement is copying product identifiers exactly from supplied text.

For example, AX-1048-B must not become AX-1084-B. An answer can read perfectly and still send someone to the wrong replacement part.

The comparison I’m planning

I would keep the source prompts, tokenizer, chat template, context limit, and decoding settings consistent, then compare a higher-precision reference with lower-bit builds. The test set would include similar-looking identifiers, leading zeros, and several product codes in the same passage.

For exact identifier copying, should I score only character-for-character matches, or also separate wrong selection from transcription errors? Choosing the wrong code from the passage seems different from inventing a code that was never there.

What else should a quantized model accuracy benchmark control for before I attribute a regression to quantization? I’m looking for a practical evaluation, not just a general language benchmark score.