Execution Planning

When a GGUF file is loaded, PureJavaBackend selects the kernels, batching strategy, cache behavior, and optional native acceleration that fit both the model and the running JVM. Those choices form an immutable execution plan used by every request. Applications can inspect the same choices through BackendDiagnostics before serving traffic.

The planner combines:

  • tensors and quantization formats present in every layer

  • structured Vectors runtime capabilities

  • JVM vendor, version, compiler identity, vector width, and processor topology

  • exact ModelJars performance profiles when a registry adapter is used

  • explicit deployment overrides

BackendDiagnostics records enabled, disabled, and unsupported choices. It also retains the exact marker coordinate, model SHA, runtime selector, and rejected profile reasons when that metadata is available.

Profile Matching

Models can execute the same mathematical operation in several correct ways. For example, it can process prompt tokens in different batch sizes, use a kernel specialized for a quantization format, or replace a measured numeric kernel with its Rust/FFM implementation. The fastest choice depends on the exact model file, CPU, JDK, and backend. A choice that improves one combination can slow down another.

A performance profile is the result of benchmarking one model artifact on a known runtime. It records the configuration that improved throughput or latency while preserving the model’s expected output. Profile matching lets Models reuse that result only when the current deployment is equivalent to the environment that produced it.

A profile is not how performance reaches a deployment. Every setting listed below whose faster route was measured byte-identical is enabled when unset, so an unmatched host gets it anyway. That was not always so: until 0.3.54 seven of these defaulted to off, and the only thing that could enable them was an exactly-matching profile — and because a selector pins an exact cpu-model, processors and usually an exact vm-version, almost no deployment matched one. A profile now carries only what a measurement found better than the default for one exact artifact, which is what it was always meant to mean.

The following fields establish that equivalence:

  1. Artifact SHA-256: identifies the exact GGUF bytes. Two files with the same model name can use different revisions, quantization formats, or tensor layouts.

  2. Runtime selector: identifies the JDK implementation and version, CPU architecture, available vector width, and other hardware characteristics that affect numeric kernel performance.

  3. Backend and tensor layout: confirms that the recommendation targets the selected Java or Rust/FFM path and the data formats present in the model.

  4. Output evidence: confirms that the optimized plan produced the expected deterministic output during qualification.

  5. JVM startup arguments: confirms that options required by the measurement, such as compiler controls, were active before the JVM started.

A missing compiler option is reported as requiring a process restart. Models does not claim to apply a JVM startup option after the process has started. Explicit deployment properties override profile recommendations. Malformed explicit values fail at load time instead of silently reverting to defaults.

Deployment Properties

Property Purpose

models.purejava.groupedProjections

Share activation preparation across compatible grouped projections

models.purejava.mixedKProjections

Enable exact mixed K-quant Q/K/V grouping

models.purejava.q4Kernel

Select widened, short-pairwise, or unsigned-pairwise Q4 arithmetic

models.purejava.q6BatchedKernel

Select one-query-block or two-query-block Q6_K prompt-prefill register tiling

models.purejava.prefillBatchSize

Set the immutable prompt-prefill batch size

models.purejava.finalLayerPrefillPruning

Restrict eligible final-layer FFN work to requested output rows

models.purejava.finalLayerKvOnlyPrefill

Produce all final-layer K/V rows while computing Q/output/FFN only where needed

models.purejava.batchedAttentionScores

Batch exact attention-score dot products. Enabled when unset since 0.3.54: measured byte-identical on Q4_0, Q8_0 and Q4_K artifacts across both backends

models.purejava.batchedAttentionValues

Batch cached value-row accumulation. Enabled when unset on the same evidence

models.purejava.stagedQuantizedFfn

Use a measured staged Q4_0/Q8_0 FFN schedule. Enabled when unset; the rust-ffm path still forces it off, because the shim owns those projections

models.purejava.stagedQuantizedLayer

Retain a wider staged quantized layer schedule. Enabled when unset, same exception

models.purejava.blockMajorQ8Activations

Store staged Q8_0 activations in block-major order. Enabled when unset

models.purejava.q8BlockMajorKernel

Select scattered, row-accumulated, or float-lane-accumulated Q8_0 arithmetic

models.purejava.parallelQ8FfnPreparation

Prepare eligible Q8_0 FFN activations in parallel. Enabled when unset; it also requires the staged layer plan and block-major activations, and reports DISABLED with that reason when either is withheld

models.purejava.maxContextLength

Impose a lower hard sequence-length limit

models.native.kernels.threads

Set the bounded native worker-context size used by qualified projection and recurrence profiles

models.native.quantizedDecode, models.native.gatedDeltaNet, models.native.q5_0.grouped and models.native.loadWarmup were removed. Each vetoed a Rust kernel the shim had already reported a capability for, and each defaulted to off unless an exactly-matching ModelJar profile supplied it — so an ordinary host ran the slow route. Sending decode to the Java path cost 6.2x on one host and harness, 2.52 against 15.72 tokens/s. The shim now serves every tensor type it reports a capability for, decode included, and the Java path remains only where there is no capability. Setting one of the removed properties now fails with a message saying so rather than being ignored.

Grouped Projection Paths

Compatible Q/K/V projections can share one activation load. Eligible mixed-K projections additionally share one Q8_K activation quantization and one row dispatch. The mixed route remains inactive for all other tensor layouts.

Batch-major prefill kernels cover Q4_0, Q5_0, Q8_0, Q4_K, Q5_K, and Q6_K. Supporting Q5_0 prevents mixed DeepSeek-Coder files from reducing the complete prefill plan to one token at a time.

The same batch-major projection kernels can advance independent decode sessions while each request retains its own position and KV cache. See Concurrent Session Batching for the serving API, lifecycle rules, and measured throughput/memory tradeoff.

Profiled Q6_K Register Tiling

Q6_K weights store six-bit values in separate low- and high-bit planes. During prompt processing, Models can reconstruct one weight vector and apply it to either one or two prompt queries before loading the next vector. The one-query form limits live SIMD state and is the conservative default. The two-query form reuses reconstructed weights and improves throughput on the qualified x86 path.

The planner accepts two-query-block only when the loaded model uses Q6_K, the prompt batch contains at least four rows, the runtime is x86, and the active Vector API species is at least 256 bits. Other runtimes retain one-query-block. An exact ModelJar performance profile can recommend the faster form; an explicit deployment property can override that recommendation. Diagnostics report the decision as q6-batched-kernel.

Batched Attention

Attention-value batching accumulates cached value rows through the generic Vectors weighted-row primitive while preserving the existing summation order. Diagnostics report this path as batched-attention-values.

Attention-score batching shares each query-vector load across two strided key-cache rows while preserving the active provider’s independent dot-product result bit for bit. Diagnostics report it as batched-attention-scores.

Both paths are disabled for ordinary loads. An exact profile or an explicit deployment property can enable them.

Staged Quantized Feed-Forward Execution

Eligible staged FFN layers require:

  • batched prefill

  • thread-shareable Q4_0 or Q8_0 gate, up, and down weights

  • parallel GGUF execution

  • the persistent Vectors row executor

The first stage runs gate/up projection, exact SwiGLU, and Q8_0 preparation. The second stage runs the down projection. Diagnostics report staged-quantized-ffn.

Retained Quantized Layer Execution

When attention output, gate, up, and down projections use Q4_0 or Q8_0, one seven-stage Vectors publication can span:

  1. Q/K/V normalization, RoPE, and cache writes.

  2. Exact grouped-query attention.

  3. Q8_0 attention preparation.

  4. Output projection.

  5. Residual addition, FFN normalization, and Q8_0 input preparation.

  6. Gate/up projection and exact SwiGLU.

  7. Down projection.

Cache storage is reserved before worker publication and each batch row owns separate attention-score scratch. Layers with another attention-output format can retain the narrower FFN-only schedule. Diagnostics report the wider route as staged-quantized-layer.

Final-Layer Prompt Pruning

Ordinary prompt prefill requests logits only for the last prompt token. For a validated final layer whose attention and FFN projections are uniformly Q4_0 or Q8_0, Models produces every K/V cache row but runs Q, attention, output projection, and FFN only for the requested output row.

Layouts that qualify only for FFN pruning retain the narrower exact path. Speculative verification and observer-backed diagnostics request all rows and therefore keep the complete path.

Set models.purejava.finalLayerKvOnlyPrefill=false to retain only FFN pruning. Set models.purejava.finalLayerPrefillPruning=false to disable both stages for rollback or controlled A/B measurement.