I’m choosing between 4-bit and 8-bit versions of the same DeepSeek model for extracting amounts, dates, and invoice numbers from plain text. Faster responses are useful, but a misplaced decimal is a bigger problem for this task than awkward wording.

What would be a fair DeepSeek quantization accuracy test? I can keep the prompt, source documents, and generation settings fixed, but I’m unsure how large the test set needs to be before a difference is meaningful.

My initial plan is to score exact field matches and separately count missing values, invented values, and decimal errors. I’d include difficult cases such as credit notes, mixed currency symbols, and invoice numbers with leading zeros.

Would you also repeat each test to measure output variability? And when comparing 4-bit and 8-bit quantization, what should I record about the model files and inference backend so I don’t accidentally attribute another configuration difference to quantization?