NVIDIA GPU Acceleration
backend-cuda optionally executes the K-quant projections and grouped-query
decode attention on a qualified NVIDIA GPU, using Models-owned Rust kernels
compiled to PTX. The model parser, tokenizer, transformer graph, KV cache,
sampling and generation loop remain inside Models. There is no external
inference server, no vendor math library and no handwritten CUDA C++.
Models ships one GPU implementation. The removed TornadoVM attempt recorded
36 failed cuLaunchKernel calls and never reached its exact-token gate. Its reported
6.49 tok/s is therefore not valid throughput for working inference. The Models CUDA
path recorded 31.26 tok/s and exact token parity on that RTX 4090/model experiment.
These results reject the tested replacement; they do not prove that Java-authored
GPU kernels cannot match it. The failed experiment remains in
benchmark-results/2026-10-07-tornado-removal.
Nothing activates it by accident
The jar carries no META-INF/services entry, so no ServiceLoader discovers
it. An application opens the accelerator explicitly:
CudaGgufBatchedMatrixKernel.Status status = CudaGgufBatchedMatrixKernel.open();
open() never throws for an absent, old or ineligible device; it returns a
status explaining why. -Dmodels.cuda.disabled=true refuses unconditionally,
and -Dmodels.cuda.attention.disabled=true ablates only the attention kernel,
counting the refusal rather than hiding it.
One module for every qualifying device
PTX is device code, so a single sm_80 module serves every device of compute
capability 8.0 or above. It travels inside the jar under
META-INF/models/cuda/ with a SHA-256 the Java loader recomputes on load, and
an ABI field that must match the Java binding. No CUDA toolkit is needed to
build or to run; only a shipping libcuda.so.1.
Measured gates
Run them with models-bench cuda-kernel-gate, which has three modes and one
report shape:
models-bench cuda-kernel-gate --mode capability --require-device true \
--report capability.json --models-revision "$(git rev-parse HEAD)"
models-bench cuda-kernel-gate --mode parity --model model.gguf \
--prompts prompts.txt --max-tokens 64 --require-device true \
--report parity.json --models-revision "$(git rev-parse HEAD)"
models-bench cuda-kernel-gate --mode decode --model model.gguf \
--prompts prompts.txt --max-tokens 64 --warmup-tokens 16 --arm both \
--require-device true --report decode.json \
--models-revision "$(git rev-parse HEAD)"
--arm both measures the accelerated arm and the Vector API control in the
same process on the same host, which is the only denominator that counts.
--models-revision requires a full 40-character SHA.
Measured on an RTX 4090 (compute capability 8.9), Granite 4.1 3B Q4_K_M, 20 prompts:
| Gate | Result | Evidence |
|---|---|---|
G1 exact token parity |
passed |
1280 of 1280 token ids identical, |
G4 decode speed |
passed |
6.184x against a 3.00x threshold, 31.26 against 5.06 tok/s,
|
G2 routing observability |
passed |
2,405,576 accelerated operations, zero declined projections |
G5 startup honesty |
passed |
344 ms readiness against a 120,000 ms ceiling |
G1 carries no tolerance. Each qualifying hardware profile and each architecture family earns its own run before the result generalises: the figures above are one device and one model.
What routes, and how to tell
The report records accelerated operations per weight format and stage, plus
declinedProjections keyed by format, stage, reason and shape. That second map
exists because an unimplemented path is not a refusal: the FFN gate and up
projections ran on the CPU through an entire measurement because
multiplyDual was missing, 53.3% of a layer’s projection arithmetic, and an
empty refusals map read as "everything ran on the device". A projection that
answers no to isEligible now increments a counter that names it.
Read launchesPerDecodeStep, transfersPerDecodeStep and
activationBytesPerDecodeStep together with the throughput. The decode path
issues roughly 321 launches and 522 transfers per token on a 40-block model,
because every projection result returns to a Java array; that is the standing
overhead and the next structural lever against it is device-resident
activations.
