Execution Planning
When a GGUF file is loaded, PureJavaBackend selects the kernels,
batching strategy, cache behavior, and optional native acceleration that
fit both the model and the running JVM. Those choices form an immutable
execution plan used by every request. Applications can inspect the same
choices through BackendDiagnostics before serving traffic.
The planner combines:
-
tensors and quantization formats present in every layer
-
structured Vectors runtime capabilities
-
JVM vendor, version, compiler identity, vector width, and processor topology
-
exact ModelJars performance profiles when a registry adapter is used
-
explicit deployment overrides
BackendDiagnostics records enabled, disabled, and unsupported choices.
It also retains the exact marker coordinate, model SHA, runtime
selector, and rejected profile reasons when that metadata is available.
Profile Matching
Models can execute the same mathematical operation in several correct ways. For example, it can process prompt tokens in different batch sizes, use a kernel specialized for a quantization format, or replace a measured numeric kernel with its Rust/FFM implementation. The fastest choice depends on the exact model file, CPU, JDK, and backend. A choice that improves one combination can slow down another.
A performance profile is the result of benchmarking one model artifact on a known runtime. It records the configuration that improved throughput or latency while preserving the model’s expected output. Profile matching lets Models reuse that result only when the current deployment is equivalent to the environment that produced it.
A profile is not how performance reaches a deployment. Every setting
listed below whose faster route was measured byte-identical is enabled
when unset, so an unmatched host gets it anyway. That was not always so:
until 0.3.54 seven of these defaulted to off, and the only thing that
could enable them was an exactly-matching profile — and because a
selector pins an exact cpu-model, processors and usually an exact
vm-version, almost no deployment matched one. A profile now carries
only what a measurement found better than the default for one exact
artifact, which is what it was always meant to mean.
The following fields establish that equivalence:
-
Artifact SHA-256: identifies the exact GGUF bytes. Two files with the same model name can use different revisions, quantization formats, or tensor layouts.
-
Runtime selector: identifies the JDK implementation and version, CPU architecture, available vector width, and other hardware characteristics that affect numeric kernel performance.
-
Backend and tensor layout: confirms that the recommendation targets the selected Java or Rust/FFM path and the data formats present in the model.
-
Output evidence: confirms that the optimized plan produced the expected deterministic output during qualification.
-
JVM startup arguments: confirms that options required by the measurement, such as compiler controls, were active before the JVM started.
A missing compiler option is reported as requiring a process restart. Models does not claim to apply a JVM startup option after the process has started. Explicit deployment properties override profile recommendations. Malformed explicit values fail at load time instead of silently reverting to defaults.
Deployment Properties
| Property | Purpose |
|---|---|
|
Share activation preparation across compatible grouped projections |
|
Enable exact mixed K-quant Q/K/V grouping |
|
Select |
|
Select |
|
Set the immutable prompt-prefill batch size |
|
Restrict eligible final-layer FFN work to requested output rows |
|
Produce all final-layer K/V rows while computing Q/output/FFN only where needed |
|
Batch exact attention-score dot products. Enabled when unset since 0.3.54: measured byte-identical on Q4_0, Q8_0 and Q4_K artifacts across both backends |
|
Batch cached value-row accumulation. Enabled when unset on the same evidence |
|
Use a measured staged
Q4_0/Q8_0 FFN schedule. Enabled when unset; the |
|
Retain a wider staged quantized layer schedule. Enabled when unset, same exception |
|
Store staged Q8_0 activations in block-major order. Enabled when unset |
|
Select |
|
Prepare eligible Q8_0 FFN
activations in parallel. Enabled when unset; it also requires the
staged layer plan and block-major activations, and reports |
|
Impose a lower hard sequence-length limit |
|
Set the bounded native worker-context size used by qualified projection and recurrence profiles |
models.native.quantizedDecode, models.native.gatedDeltaNet,
models.native.q5_0.grouped and models.native.loadWarmup were removed.
Each vetoed a Rust kernel the shim had already reported a capability for,
and each defaulted to off unless an exactly-matching ModelJar profile
supplied it — so an ordinary host ran the slow route. Sending decode to
the Java path cost 6.2x on one host and harness, 2.52 against 15.72
tokens/s. The shim now serves every tensor type it reports a capability
for, decode included, and the Java path remains only where there is no
capability. Setting one of the removed properties now fails with a message
saying so rather than being ignored.
Grouped Projection Paths
Compatible Q/K/V projections can share one activation load. Eligible mixed-K projections additionally share one Q8_K activation quantization and one row dispatch. The mixed route remains inactive for all other tensor layouts.
Batch-major prefill kernels cover Q4_0, Q5_0, Q8_0, Q4_K, Q5_K, and Q6_K. Supporting Q5_0 prevents mixed DeepSeek-Coder files from reducing the complete prefill plan to one token at a time.
The same batch-major projection kernels can advance independent decode sessions while each request retains its own position and KV cache. See Concurrent Session Batching for the serving API, lifecycle rules, and measured throughput/memory tradeoff.
Profiled Q6_K Register Tiling
Q6_K weights store six-bit values in separate low- and high-bit planes. During prompt processing, Models can reconstruct one weight vector and apply it to either one or two prompt queries before loading the next vector. The one-query form limits live SIMD state and is the conservative default. The two-query form reuses reconstructed weights and improves throughput on the qualified x86 path.
The planner accepts two-query-block only when the loaded model uses
Q6_K, the prompt batch contains at least four rows, the runtime is x86,
and the active Vector API species is at least 256 bits. Other runtimes
retain one-query-block. An exact ModelJar performance profile can
recommend the faster form; an explicit deployment property can override
that recommendation. Diagnostics report the decision as
q6-batched-kernel.
Batched Attention
Attention-value batching accumulates cached value rows through the
generic Vectors weighted-row primitive while preserving the existing
summation order. Diagnostics report this path as
batched-attention-values.
Attention-score batching shares each query-vector load across two
strided key-cache rows while preserving the active provider’s
independent dot-product result bit for bit. Diagnostics report it as
batched-attention-scores.
Both paths are disabled for ordinary loads. An exact profile or an explicit deployment property can enable them.
Staged Quantized Feed-Forward Execution
Eligible staged FFN layers require:
-
batched prefill
-
thread-shareable Q4_0 or Q8_0 gate, up, and down weights
-
parallel GGUF execution
-
the persistent Vectors row executor
The first stage runs gate/up projection, exact SwiGLU, and Q8_0
preparation. The second stage runs the down projection. Diagnostics
report staged-quantized-ffn.
Retained Quantized Layer Execution
When attention output, gate, up, and down projections use Q4_0 or Q8_0, one seven-stage Vectors publication can span:
-
Q/K/V normalization, RoPE, and cache writes.
-
Exact grouped-query attention.
-
Q8_0 attention preparation.
-
Output projection.
-
Residual addition, FFN normalization, and Q8_0 input preparation.
-
Gate/up projection and exact SwiGLU.
-
Down projection.
Cache storage is reserved before worker publication and each batch row
owns separate attention-score scratch. Layers with another
attention-output format can retain the narrower FFN-only schedule.
Diagnostics report the wider route as staged-quantized-layer.
Final-Layer Prompt Pruning
Ordinary prompt prefill requests logits only for the last prompt token. For a validated final layer whose attention and FFN projections are uniformly Q4_0 or Q8_0, Models produces every K/V cache row but runs Q, attention, output projection, and FFN only for the requested output row.
Layouts that qualify only for FFN pruning retain the narrower exact path. Speculative verification and observer-backed diagnostics request all rows and therefore keep the complete path.
Set models.purejava.finalLayerKvOnlyPrefill=false to retain only FFN
pruning. Set models.purejava.finalLayerPrefillPruning=false to disable
both stages for rollback or controlled A/B measurement.
