RigorMed
← Research resources

Computer science and AI

How to report a reproducible inference benchmark

Define the workload and the timing boundaries before comparing latency or throughput. Offline batches and online requests answer different questions and need separate settings and metrics.

Computer science and AI

Gather model weights, data version, inference code, hardware details, and logs. Decide whether the claim concerns latency under fixed load or sustainable throughput under a fixed latency budget.

MLPerf defines metrics and rules by scenario. This worksheet is for an ordinary research report. A formal MLPerf submission uses the relevant rules and tooling.

How to do it

  1. Describe the comparison conditions

    Keep the task and input-length distribution visible. Record concurrency, batch size, warmup, caching, hardware, drivers, precision, and framework versions.

  2. Specify timing boundaries

    Say whether timing runs from client dispatch to the full response, covers server execution, or measures one operator. Document synchronization for asynchronous GPU work. For generation, record input and output lengths and consider time to first token and generation throughput separately.

  3. Check quality and failed requests

    Measure quality on the same dataset with the same metric. Record failures, timeouts, and retries. Define exclusions in advance and retain request logs.

  4. Report repeats and point back to raw data

    Give the request count, the number of repeated runs, and summary statistics such as p50 and p95. Attach configuration and run IDs so each table row can be traced to a log.

Worked example

This synthetic API benchmark uses the same device and request stream with end-to-end timing.

Worked example
MeasureBaseline AMethod B
Requests and concurrency1,000 per run, concurrency 81,000 per run, concurrency 8
End-to-end p50 / p9517 / 35 ms15 / 42 ms
Throughput and timeouts112 requests/s; 2 timeouts118 requests/s; 7 timeouts
QualityAccuracy 0.91, test set v2Accuracy 0.91, test set v2

B lowers median latency while raising tail latency and timeouts. B misses a prespecified 40 ms p95 budget in this example. A single run gives limited evidence of a stable gain.

Common questions

Why report latency and throughput together?

Larger batches or higher concurrency may raise throughput and increase waiting time. Report both under the same load.

Can I compare runs on different hardware?

You can compare complete systems and disclose the hardware. A claim about the algorithm itself needs additional controlled evidence.

Verified sources

The official sources below were checked on 2026-11-13. Access conditions and update dates are rechecked before publication.

  1. MLCommons — MLPerf Inference
  2. MLCommons — Inference Datacenter benchmarks

RigorMed workspace

Prepare a repeatable inference report

Bring two configurations, raw timing logs, and quality metrics. Organize the comparison conditions, timing definitions, and claims that need checking.