Computer science and AI
How to report a reproducible inference benchmark
Define the workload and the timing boundaries before comparing latency or throughput. Offline batches and online requests answer different questions and need separate settings and metrics.
Gather model weights, data version, inference code, hardware details, and logs. Decide whether the claim concerns latency under fixed load or sustainable throughput under a fixed latency budget.
MLPerf defines metrics and rules by scenario. This worksheet is for an ordinary research report. A formal MLPerf submission uses the relevant rules and tooling.
How to do it
Describe the comparison conditions
Keep the task and input-length distribution visible. Record concurrency, batch size, warmup, caching, hardware, drivers, precision, and framework versions.
Specify timing boundaries
Say whether timing runs from client dispatch to the full response, covers server execution, or measures one operator. Document synchronization for asynchronous GPU work. For generation, record input and output lengths and consider time to first token and generation throughput separately.
Check quality and failed requests
Measure quality on the same dataset with the same metric. Record failures, timeouts, and retries. Define exclusions in advance and retain request logs.
Report repeats and point back to raw data
Give the request count, the number of repeated runs, and summary statistics such as p50 and p95. Attach configuration and run IDs so each table row can be traced to a log.
Worked example
This synthetic API benchmark uses the same device and request stream with end-to-end timing.
| Measure | Baseline A | Method B |
|---|---|---|
| Requests and concurrency | 1,000 per run, concurrency 8 | 1,000 per run, concurrency 8 |
| End-to-end p50 / p95 | 17 / 35 ms | 15 / 42 ms |
| Throughput and timeouts | 112 requests/s; 2 timeouts | 118 requests/s; 7 timeouts |
| Quality | Accuracy 0.91, test set v2 | Accuracy 0.91, test set v2 |
B lowers median latency while raising tail latency and timeouts. B misses a prespecified 40 ms p95 budget in this example. A single run gives limited evidence of a stable gain.
Common questions
Why report latency and throughput together?
Larger batches or higher concurrency may raise throughput and increase waiting time. Report both under the same load.
Can I compare runs on different hardware?
You can compare complete systems and disclose the hardware. A claim about the algorithm itself needs additional controlled evidence.
Verified sources
The official sources below were checked on 2026-11-13. Access conditions and update dates are rechecked before publication.
RigorMed workspace
Prepare a repeatable inference report
Bring two configurations, raw timing logs, and quality metrics. Organize the comparison conditions, timing definitions, and claims that need checking.
