inferlint measures what your inference server actually did—and flags what it didn't tell you.

For vLLM, SGLang and TensorRT-LLM. Around a benchmark it reads the server's boot log and counters, measures achieved concurrency, preemptions, KV-cache use, GPU state and timing, and checks the run for 15 known problems. It tells the engine from what it reads, and works with the load tool you already use.

• 10 of 32requests ran at once; the benchmark said 32 (T8)
• 22preemptions, and no line about them in the server log (T1)
• 10 → 7the most that fit, falling as answers grew (T9)

One command around your benchmark

# install
pipx install inferlint

# vLLM: start the server, record it while the benchmark runs, check it, stop it
inferlint xray -o run/ --serve "vllm serve MODEL" -- vllm bench serve --model MODEL --max-concurrency 32 ...

# SGLang: its metrics need --enable-metrics
inferlint xray -o run/ --serve "sglang serve --model-path MODEL --enable-metrics"     -- python -m sglang.benchmark.serving --model MODEL --max-concurrency 32 ...

# TensorRT-LLM: its metrics need return_perf_metrics: true and enable_iter_perf_stats: true in the --config file
inferlint xray -o run/ --serve "trtllm-serve serve MODEL --config llm_api.yaml"     -- python -m tensorrt_llm.serve.scripts.benchmark_serving --model MODEL --max-concurrency 32 ...

# a team on one engine can name it once: vllm, sglang or trtllm
export INFERLINT_ENGINE=sglang

== engine: vLLM (from the --serve command)
== is the GPU free?
[ T2] PASS  2 consecutive clear readings
== start the server (output in run/boot.log)
== boot log
[T12] PASS  no failure in log
[ T5] WARN  attention block size forced to 784 tokens (requested 16); each request's KV is allocated in 784-token units
[ T6] PASS  nothing to check: the server was started without --attention-backend, so vLLM chose its own (FLASH_ATTN) and nothing could be ignored; there was no draft model either
== probe: one request, then 8 at once
[ T7] PASS  1 + 8 concurrent requests served, server alive
== load: vllm bench serve --model MODEL ... --max-concurrency 32 --save-result --result-dir run --result-filename bench.json
… the benchmark's own output
== checks
[ T1] TRIPWIRE-FAILED  21 preemptions during the run
[T10] PASS  316.82 tokens/s over 101.005s; an integer-second timer would say 316.83 (+0.01%)
[T14] PASS  gauge denominator is 42 = num_gpu_blocks (43) - 1 null block
[ T9] WARN  ceiling moved between 6 and 10 as requests grew (one request held 9.5% to 16.3% of the cache); concurrency was not a constant of this run
[ T8] TRIPWIRE-FAILED  requested 32 concurrent, server never ran more than 10
[T15] PASS  client and server agree: 32 requests, 32,000 output tokens
== stop the server
[ T2] PASS  2 consecutive clear readings
== what limited this run
- The server ran fewer requests at once than the load sent. The load kept 32 requests open, but the server ran at most 10 at once. It also preempted 21 running requests.
- Requests waited long to start. Half the requests waited more than 28.7 s for their first token, the slowest (p99) 73.2 s, most of it in the queue.
- On average 8 of 32 requests ran at once. The load kept 32 requests open; the server's recording shows 8.0 running on average while it was busy. The load tool measured 337 output tokens/s over the whole run.
12 checks (8 pass, 2 warn, 2 tripwire-failed)
report: run/report.html
files:  run

The output of the vLLM command on vLLM 0.28, from another run of the test in the chart, made with inferlint 0.3.0 (its last finding as 0.3.1 words it), with paths and the load command shortened. Tested live on vLLM 0.28, 0.29 and 0.30, SGLang 0.5.20 and TensorRT-LLM 1.3.0rc29, on an RTX 4090. No GPU at hand? The tutorial runs every check on the recorded files in two minutes.