Tutorial

inferlint, step by step

inferlint measures what your inference server actually did—and flags what it didn't tell you. From the start-up log and counters of a vLLM, SGLang or TensorRT-LLM server it measures achieved concurrency, preemptions, KV-cache use, GPU state and timing, and checks the run for 15 known problems. Each check answers Pass, Warning, Fail or Can't tell, or says it does not apply to the engine. It works alongside whatever load tool you already use. The examples on this page all come from one recorded run: vLLM 0.28 on an RTX 4090, a 27B model, 32 requests at once, 1,000 output tokens each. The commands are the same for the other two engines; what they need and how they differ is below.

Asked for 32 at once. The server ran 10.

Requests in flight during the recorded run, read every 0.25 s

Hover over the chart, or focus it and use the arrow keys, to read any moment.

Show the numbers, every 5 seconds

10 of 32requests ran at once. The benchmark said 32. T8
22preemptions. The server log mentions none. T1
10 → 7the most that fit, falling as answers grew. T9

Quick start · 2 minutes · no GPU

Try it on the recorded run

The repository includes the files from the recorded run, and from recorded SGLang and TensorRT-LLM runs, so you can run every check on any computer, Linux, macOS or Windows, without a GPU or a server.

1Install

pipx install inferlint
inferlint --help

pipx gives inferlint its own environment and puts the inferlint command on your PATH. There is nothing to activate, and it works from any folder. Plain pip install inferlint works too. Needs Python 3.10 or newer; inferlint has no other dependencies.

2Get the recorded run

git clone https://github.com/radianvector/inferlint
cd inferlint

3Run the checks on it

First, a short name for the folder with the recorded files:

L=tests/fixtures/vllm-0.28/live

$L = "tests\fixtures\vllm-0.28\live"

On Windows, everything here works in PowerShell. teardown needs Linux, because it finds the server's processes the way Linux lists them. gpu-inspect shows the GPU on Windows too, but its free-card check (T2) needs Linux, so there it answers Can't tell. Run those two in WSL, where the engines run anyway.

The idea behind most of the checks

Why only 10? The GPU memory, in 3D

While a request is being answered, the server keeps its working notes in GPU memory, called the KV cache. vLLM hands that memory out in blocks. In this run there were 43 blocks of 784 tokens each, and one is held back, so 42 were usable (T14).

Each request needed 4 blocks to start, so 10 fit and the other 22 waited. As answers grew past 784 tokens they needed more blocks, and fewer fit. Press Play to replay the recorded run, or drag the slider through it.

Seconds into the run0.0

Running0

Waiting0

Blocks each, average–

Finished0

Preemptions so far0

Time in the recorded run

0 s 16 s: first preemption 93 s

The 42 usable blocks0 in use, 42 free

Each tower is a running request and each tile a waiting one; a tile with an outline has just been pushed back. Outlined spots are places with no room while requests wait. The recording has the total number of blocks in use, not each request's count, so the towers share that total evenly.

Check your own run

A GPU run with one command

You need Linux or WSL, an NVIDIA GPU, the engine (vLLM, SGLang or TensorRT-LLM), and inferlint installed on the same machine (step 1 above). inferlint xray starts the server, checks it, records it while your benchmark runs, writes the report and stops the server. Your load tool goes after --; inferlint sends no load of its own.

inferlint xray -o run/ --serve "vllm serve $MODEL --max-model-len 16384" \
    -- vllm bench serve --model "$MODEL" --dataset-name random \
       --random-input-len 88 --random-output-len 1000 --ignore-eos \
       --num-prompts 32 --max-concurrency 32

You'll see (vLLM 0.28 on the RTX 4090 used for this page, shortened)

== engine: vLLM (from the --serve command)
== is the GPU free?
[ T2] PASS  2 consecutive clear readings
== start the server (output in run/boot.log)
== boot log
[T12] PASS  no failure in log
[ T5] WARN  attention block size forced to 784 tokens (requested 16); each request's KV is allocated in 784-token units
[ T6] PASS  nothing to check: the server was started without --attention-backend, so vLLM chose its own (FLASH_ATTN) and nothing could be ignored; there was no draft model either
== probe: one request, then 8 at once
[ T7] PASS  1 + 8 concurrent requests served, server alive
== load: vllm bench serve --model MODEL ... --max-concurrency 32 --save-result --result-dir run --result-filename bench.json
…
== checks
[ T1] TRIPWIRE-FAILED  21 preemptions during the run
[T10] PASS  316.82 tokens/s over 101.005s; an integer-second timer would say 316.83 (+0.01%)
[T14] PASS  gauge denominator is 42 = num_gpu_blocks (43) - 1 null block
[ T9] WARN  ceiling moved between 6 and 10 as requests grew (one request held 9.5% to 16.3% of the cache); concurrency was not a constant of this run
[ T8] TRIPWIRE-FAILED  requested 32 concurrent, server never ran more than 10
[T15] PASS  client and server agree: 32 requests, 32,000 output tokens
== stop the server
[ T2] PASS  2 consecutive clear readings
== what limited this run
- The server ran fewer requests at once than the load sent. The load kept 32 requests open, but the server ran at most 10 at once. It also preempted 21 running requests.
- Requests waited long to start. Half the requests waited more than 28.7 s for their first token, the slowest (p99) 73.2 s, most of it in the queue.
- On average 8 of 32 requests ran at once. The load kept 32 requests open; the server's recording shows 8.0 running on average while it was busy. The load tool measured 337 output tokens/s over the whole run.
12 checks (8 pass, 2 warn, 2 tripwire-failed)
report: run/report.html
files:  run

Above, paths are shortened, the load command is cut, and the benchmark's own output is left out (the … line). This was another run of the same test as the recorded one, made with inferlint 0.3.0 (its last finding as 0.3.1 words it), so its counts differ a little: 21 preemptions instead of 22, and a lowest ceiling of 6 instead of 7.

Every file lands in run/: the boot log, the before and after readings, the recording, the benchmark's own result file and report.html. The server is stopped at the end even if a step fails or you press Ctrl-C. To check a server that is already running, leave out --serve and give --url and --boot-log.

For each engine's own benchmark client (vllm bench serve, SGLang's python -m sglang.benchmark.serving, TensorRT-LLM's python -m tensorrt_llm.serve.scripts.benchmark_serving), xray asks the client to save its result so it can compare the benchmark's own counts with the server's (T15), and reads --max-concurrency as the concurrency asked for (T8). Any other load tool runs as it is.

Or step by step

The same routine as separate commands, for scripts that control each step. Use two terminals in the same folder: A runs the server, B runs everything else. The picture shows what runs when; click a number to open that step.

Before you start
1Check that the GPU is freeinferlint gpu-inspect
A server from an earlier test can still be holding the card. In the recorded run, a killed server's engine kept 21.3 GiB after its front end was gone. A new server started then would only get the memory that was left.
```
inferlint gpu-inspect || exit 1
```
You'll see (the RTX 4090 used for this page)
```
GPU 0   NVIDIA GeForce RTX 4090
        Ada Lovelace, compute capability 8.9, driver 616.56
        memory: 24,564 MiB total, 0 MiB used, 24,138 MiB free
[ T2] PASS  2 consecutive clear readings
```
It shows which GPU this is and its memory, then checks that no server process is holding it. It only looks; it never stops anything. It exits with 1 if the card is not clear, so a script stops there.

Checks T2
Start the server
2Start the server and keep its logvllm serve … | tee boot.log
In terminal A, start the server with your usual command and save everything it prints at start-up to a file. The log records how much memory the server set aside and how it divided it; inferlint reads it in the next step. Use a new log file for each start.
```
vllm serve "$MODEL" --max-model-len 16384 2>&1 | tee boot.log
# or SGLang, with its metrics on
sglang serve --model-path "$MODEL" --enable-metrics 2>&1 | tee boot.log
# or TensorRT-LLM, with return_perf_metrics: true and enable_iter_perf_stats: true in llm_api.yaml
trtllm-serve serve "$MODEL" --config llm_api.yaml 2>&1 | tee boot.log
```
Wait for Application startup complete; it took 2 min 4 s in the recorded run. Run the remaining steps in terminal B.
3Read the start-up loginferlint check-log boot.log
This checks how the server set itself up: whether start-up failed and why, how large its memory blocks are, and whether it used the attention method you asked for.
```
inferlint check-log boot.log
```
You'll see (recorded run)
```
[T12] PASS  no failure in log
[ T5] WARN  attention block size forced to 784 tokens (requested 16); each request's KV is allocated in 784-token units
[ T6] PASS  nothing to check: the server was started without --attention-backend, so vLLM chose its own (FLASH_ATTN) and nothing could be ignored; there was no draft model either
```
With two or more logs from the same settings, check-log also compares them (T4, T11). inferlint boot-facts boot.log lists everything it read from the log.

Checks T12 T5 T6 T4 T11
4Send it a few real requestsinferlint probe URL
Some settings start cleanly and then crash on the first request. The probe sends one request, then 8 at once, and checks that the server is still running afterwards.
```
inferlint probe http://127.0.0.1:8000
```
Use the server's address. vLLM and TensorRT-LLM listen on port 8000 by default, SGLang on 30000.

You'll see
```
[ T7] PASS  1 + 8 concurrent requests served, server alive
```
Checks T7
Measure
5Take a reading before the testinferlint snapshot URL -o before.json
This saves all of the server's counters, with the time, to a file: tokens generated so far, requests finished, preemptions and more. Step 8 takes a second reading, and the difference between the two is what happened during your test.
```
inferlint snapshot http://127.0.0.1:8000 -o before.json
```
6Start recordinginferlint watch URL … &
Two readings show totals. To see what happened in between (how many requests ran, how many waited, how full the memory was), record the server every quarter second while the test runs. The & runs it in the background.
```
inferlint watch http://127.0.0.1:8000 -o run.series.jsonl --interval 0.25 &
```
That recording is what the chart at the top of this page is drawn from.

7Run your load testyour usual client

Use whatever benchmark you normally use. inferlint doesn't need to know which: its numbers come from the server's own counters. Note how many requests your tool keeps open at once; you'll pass that number as --requested. For example, with each engine's own client:

# vLLM
vllm bench serve --model "$MODEL" --dataset-name random \
    --random-input-len 88 --random-output-len 1000 --ignore-eos \
    --num-prompts 32 --max-concurrency 32
# SGLang
python -m sglang.benchmark.serving --model "$MODEL" --dataset-name random-ids \
    --random-input-len 88 --random-output-len 1000 --random-range-ratio 1 \
    --num-prompts 32 --max-concurrency 32
# TensorRT-LLM
python -m tensorrt_llm.serve.scripts.benchmark_serving --model "$MODEL" \
    --dataset-name random --random-ids --random-input-len 88 --random-output-len 1000 \
    --ignore-eos --num-prompts 32 --max-concurrency 32

Option names change between releases; each client's --help lists yours.

8Take a reading after, and stop recordinginferlint snapshot … -o after.json
```
inferlint snapshot http://127.0.0.1:8000 -o after.json
kill %1        # stops the watch from step 6
```
You now have four files: boot.log, before.json, after.json and run.series.jsonl. Everything from here on works from these files, on any computer, with no GPU.
Read the results

9Check what happened during the runpreemption · rate · series

inferlint preemption before.json after.json
inferlint rate before.json after.json
inferlint series run.series.jsonl --snapshot after.json --requested 32

You'll see (recorded run)

[ T1] TRIPWIRE-FAILED  22 preemptions during the run
[T10] PASS  341.44 tokens/s over 93.720s; an integer-second timer would say 344.09 (+0.77%)
[T14] PASS  gauge denominator is 42 = num_gpu_blocks (43) - 1 null block
[ T9] WARN  ceiling moved between 7 and 10 as requests grew (one request held 9.5% to 14.3% of the cache); concurrency was not a constant of this run
[ T8] TRIPWIRE-FAILED  requested 32 concurrent, server never ran more than 10

The server finished all 32 requests without errors. The two failed tripwires are about the measurement: at most 10 requests ran at once, and 22 were paused and restarted along the way.

TensorRT-LLM counts no pauses; it logs each one. On TensorRT-LLM, add --boot-log boot.log to preemption, so it counts them from the log, which tee keeps writing through the run.

Checks T1 T10 T14 T9 T8

10Make the reportinferlint report -o run.html …

One HTML page with everything. It opens with what limited the run, in up to three sentences, then gives a plain-English account of the run, every verdict, charts of what the server did, and the start-up facts. It needs no internet connection, so you can attach it to a ticket as it is.

inferlint report -o run.html --requested 32 --boot-log boot.log \
    --before before.json --after after.json --series run.series.jsonl

You'll see

[ T1] TRIPWIRE-FAILED  22 preemptions during the run
…
What limited this run:
  - The server ran fewer requests at once than the load sent. The load kept 32 requests open, but the server ran at most 10 at once. It also preempted 22 running requests.
  - Requests waited in the queue. A request waited 29.3 s on average before the server started it.
  - On average 8 of 32 requests ran at once. …
8 checks (4 pass, 2 warn, 2 tripwire-failed) -> run.html

Open run.html in any browser. The report for the recorded run shows what you get. For a folder that xray wrote, inferlint report run/ is enough: it finds the files by name.

Clean up
11Stop the server properlyinferlint teardown
The usual pkill -f "vllm serve" misses vLLM's engine process, which renames itself VLLM::EngineCore and keeps the GPU memory. teardown finds every server process, including SGLang's and TensorRT-LLM's workers, stops them, and waits until two readings in a row show the card is free. With an engine named (--engine or INFERLINT_ENGINE) it stops only that engine's servers.
```
inferlint teardown --dry-run   # only lists what it would stop
inferlint teardown
```
You'll see (after the server was killed with pkill -9)
```
target  pid=1030     VLLM::EngineCore
(pkill -f 'vllm serve' would miss 1 and wrongly hit 0)
[ T2] PASS  2 consecutive clear readings
```
Checks T2 T3

Other engines

SGLang and TensorRT-LLM

Every command works the same way on all three engines. inferlint tells the engine from the server command, its metrics and its log. Tested live on vLLM 0.28, 0.29 and 0.30, SGLang 0.5.20 and TensorRT-LLM 1.3.0rc29, on the same RTX 4090.

What each needs. SGLang serves its metrics only with --enable-metrics. TensorRT-LLM serves them only with return_perf_metrics: true in the YAML file given to trtllm-serve --config, at /prometheus/metrics, where inferlint finds them; its gauges of running and waiting requests and KV use also need enable_iter_perf_stats: true. SGLang listens on port 30000 unless told otherwise, and xray looks there.

# SGLang
inferlint xray -o run/ --serve "sglang serve --model-path $MODEL --enable-metrics" \
    -- python -m sglang.benchmark.serving --model "$MODEL" --dataset-name random-ids \
       --random-input-len 88 --random-output-len 1000 --random-range-ratio 1 \
       --num-prompts 32 --max-concurrency 32

# TensorRT-LLM
printf 'return_perf_metrics: true\nenable_iter_perf_stats: true\n' > llm_api.yaml
inferlint xray -o run/ --serve "trtllm-serve serve $MODEL --config llm_api.yaml" \
    -- python -m tensorrt_llm.serve.scripts.benchmark_serving --model "$MODEL" \
       --dataset-name random --random-ids --random-input-len 88 --random-output-len 1000 \
       --ignore-eos --num-prompts 32 --max-concurrency 32

A team on one engine can name it once: vllm, sglang or trtllm (the names as the projects write them, such as TensorRT-LLM, work too, in any case). Without a name, inferlint tells the engine from each input. With one, an input from another engine, such as a snapshot from a server on the wrong port, is then refused, and teardown stops only that engine's servers.

export INFERLINT_ENGINE=sglang     # vllm, sglang or trtllm; or --engine sglang on any command

Where they differ.

SGLang calls a preemption a retraction, counts it and logs a warning for each. TensorRT-LLM pauses requests for recompute and counts none; it logs each pause, so inferlint counts them from its log saved until after the run (T1).
SGLang can lower its running limit below --max-running-requests for hybrid models, and says so once in its log (T8).
TensorRT-LLM updates its gauges only when a request completes, so a recording can miss a peak; inferlint says "can't tell" instead of guessing (T8, T9).
Both add a request's output tokens to their counter only when it finishes, so the report leaves out the tokens-per-second chart for them and says why.
The large KV blocks of T5 and the reserved block of T14 are vLLM's.

The same model on all three engines, and every difference in detail: docs/engines.md.

Reading a result

[ T1]which check TRIPWIRE-FAILEDthe verdict 22 preemptions during the runwhat it found, in the run's own numbers

Verdict	What it means	What to do
Tripwire failed	The tripwire caught its problem in this run. For T7 and T12 the server itself failed. For every other tripwire the server worked, but a number from the run (concurrency, speed or delay) does not mean what it seems.	Fix the setup and run again, or report the number together with the finding.
Warning	Nothing is wrong on its own, but a setting or an effect changes how the numbers should be read.	Keep the warning with the results.
Can't tell	A counter or log line the check needs was missing. It never counts as a pass. The terminal shows it as `????`.	Supply the missing file, or check the engine's version.
Pass	The check looked for its problem in this run and did not find it.	Nothing. A pass covers this run only.
Does not apply	The tripwire is about a behaviour this engine does not have, such as vLLM's reserved KV block (T14) on SGLang or TensorRT-LLM. The terminal shows it as `N/A`.	Nothing.

In scripts

Every command ends with an exit code a script can test:

`0`	everything passed (warnings allowed)
`1`	a tripwire failed
`2`	a tripwire could not decide, or an input came from another engine than the one named

A tripwire that does not apply to the engine (N/A) does not change the exit code. --strict turns warnings into failures. --json prints the results as JSON, including the evidence behind each one.

The 15 tripwires

Grouped by when in a test they apply. Open one for what goes wrong, why it changes your numbers, what differs on SGLang and TensorRT-LLM, and the command that checks it. inferlint explain T9 (or inferlint explain falling-ceiling) prints a technical version in the terminal.

Cheat sheet

When	Command	Checks
The whole run, one command	`inferlint xray -o run/ --serve "vllm serve …" -- vllm bench serve …`	all that apply
Before starting a server	`inferlint gpu-inspect`	T2
After start-up	`inferlint check-log boot.log`	T5 T6 T12 (T4 T11 with 2+ logs)
After start-up	`inferlint boot-facts boot.log`	lists what the log says
Before the test	`inferlint probe URL`	T7
Before and after the test	`inferlint snapshot URL -o before.json`	saves the counters
During the test	`inferlint watch URL -o run.series.jsonl &`	records the run
After the test	`inferlint preemption before.json after.json`	T1
After the test	`inferlint rate before.json after.json`	T10
After the test	`inferlint series run.series.jsonl --snapshot after.json --requested N`	T8 T9 T14
Any time after	`inferlint report -o run.html …`	all that apply
Any time after, for a folder from xray	`inferlint report run/`	all that apply
Stopping the server	`inferlint teardown`	T2 T3
Any time	`inferlint explain T1`	explains a check
Once, for a team on one engine	`export INFERLINT_ENGINE=sglang`	refuses another engine's files