NVIDIA GPU Acceleration

backend-cuda optionally executes the K-quant projections and grouped-query decode attention on a qualified NVIDIA GPU, using Models-owned Rust kernels compiled to PTX. The model parser, tokenizer, transformer graph, KV cache, sampling and generation loop remain inside Models. There is no external inference server, no vendor math library and no handwritten CUDA C++.

Models ships one GPU implementation. The removed TornadoVM attempt recorded 36 failed cuLaunchKernel calls and never reached its exact-token gate. Its reported 6.49 tok/s is therefore not valid throughput for working inference. The Models CUDA path recorded 31.26 tok/s and exact token parity on that RTX 4090/model experiment. These results reject the tested replacement; they do not prove that Java-authored GPU kernels cannot match it. The failed experiment remains in benchmark-results/2026-10-07-tornado-removal.

Nothing activates it by accident

The jar carries no META-INF/services entry, so no ServiceLoader discovers it. An application opens the accelerator explicitly:

CudaGgufBatchedMatrixKernel.Status status = CudaGgufBatchedMatrixKernel.open();

open() never throws for an absent, old or ineligible device; it returns a status explaining why. -Dmodels.cuda.disabled=true refuses unconditionally, and -Dmodels.cuda.attention.disabled=true ablates only the attention kernel, counting the refusal rather than hiding it.

One module for every qualifying device

PTX is device code, so a single sm_80 module serves every device of compute capability 8.0 or above. It travels inside the jar under META-INF/models/cuda/ with a SHA-256 the Java loader recomputes on load, and an ABI field that must match the Java binding. No CUDA toolkit is needed to build or to run; only a shipping libcuda.so.1.

Measured gates

Run them with models-bench cuda-kernel-gate, which has three modes and one report shape:

models-bench cuda-kernel-gate --mode capability --require-device true \
  --report capability.json --models-revision "$(git rev-parse HEAD)"

models-bench cuda-kernel-gate --mode parity --model model.gguf \
  --prompts prompts.txt --max-tokens 64 --require-device true \
  --report parity.json --models-revision "$(git rev-parse HEAD)"

models-bench cuda-kernel-gate --mode decode --model model.gguf \
  --prompts prompts.txt --max-tokens 64 --warmup-tokens 16 --arm both \
  --require-device true --report decode.json \
  --models-revision "$(git rev-parse HEAD)"

--arm both measures the accelerated arm and the Vector API control in the same process on the same host, which is the only denominator that counts. --models-revision requires a full 40-character SHA.

Measured on an RTX 4090 (compute capability 8.9), Granite 4.1 3B Q4_K_M, 20 prompts:

Gate Result Evidence

G1 exact token parity

passed

1280 of 1280 token ids identical, benchmark-results/2026-10-07-g1-parity

G4 decode speed

passed

6.184x against a 3.00x threshold, 31.26 against 5.06 tok/s, benchmark-results/2026-10-07-g4-dualpath

G2 routing observability

passed

2,405,576 accelerated operations, zero declined projections

G5 startup honesty

passed

344 ms readiness against a 120,000 ms ceiling

G1 carries no tolerance. Each qualifying hardware profile and each architecture family earns its own run before the result generalises: the figures above are one device and one model.

What routes, and how to tell

The report records accelerated operations per weight format and stage, plus declinedProjections keyed by format, stage, reason and shape. That second map exists because an unimplemented path is not a refusal: the FFN gate and up projections ran on the CPU through an entire measurement because multiplyDual was missing, 53.3% of a layer’s projection arithmetic, and an empty refusals map read as "everything ran on the device". A projection that answers no to isEligible now increments a counter that names it.

Read launchesPerDecodeStep, transfersPerDecodeStep and activationBytesPerDecodeStep together with the throughput. The decode path issues roughly 321 launches and 522 transfers per token on a 40-block model, because every projection result returns to a Java array; that is the standing overhead and the next structural lever against it is device-resident activations.