A single request to my locally hosted DeepSeek distill model feels usable. When several people submit prompts together, the wait before the first token increases sharply, even though the server isn’t crashing or reporting an out-of-memory error.

I’m keeping the model, quantization, and GPU unchanged. The workload is mixed: some requests are short questions, while others include several pages of context.

Which measurements would separate the bottlenecks?

I can record time to first token, generation speed, total request duration, and GPU memory use. I’m less sure how to tell whether requests are waiting in a queue, competing during prompt processing, or being limited by memory available for active sequences.

For a useful concurrent inference benchmark, would you first use identical prompt lengths and output limits, then introduce the mixed workload? I’m also wondering whether to compare synchronized bursts with requests arriving steadily over time.

The practical goal is to set a sensible concurrency limit for a small internal service. Aggregate tokens per second alone doesn’t tell me whether someone waits an unacceptable amount of time for a short answer. What server settings and latency percentiles would you include in the test report?