In-process inference - Java 25

Models

Run open-weight text and speech models inside your Java application, without Python, a separate model server, or a network request.

Models loads GGUF, supported Safetensors, and CACT artifacts and owns tokenization, the transformer graph, KV state, sampling, and streaming. Both backends use the Java 25 runtime and Vector API. backend-java executes every kernel in Java; backend-native substitutes only selected measured bottleneck kernels with Models-owned Rust through FFM.

// Generated reference to a qualified artifact

import static org.modeljars.catalog.Qwen3_0_6b_Q4_0.MODEL;

try (var runtime = ModelJars.openRuntime(MODEL)) {
    var prompt = runtime.chatTemplate().render(messages);
    String answer = runtime.model().generate(prompt, options);
}

// Exact Maven-style marker coordinate

try (var runtime = ModelJars.openRuntime(
        "org.modeljars.huggingface:" +
        "ggml-org.qwen3-0.6b-gguf.q4_0:" +
        "3.0.0-q4_0.1")) {
    var prompt = runtime.chatTemplate().render(messages);
    String answer = runtime.model().generate(prompt, options);
}

// Apple's on-device SystemLanguageModel

try (var client = AppleFoundationModels.create()) {
    if (client.availability().available()) {
        String answer = client.generate(prompt).text();
    }
}

Read the documentation View source

runtime

Run models directly inside your Java application

GGUF, supported Hugging Face Safetensors bundles, and CACT package model structure, tokenizer metadata, and trained weights in different ways. Models maps those weights, executes the graph, maintains its attention state, and generates tokens in process. Backend selection changes only kernels justified by exact artifact and host evidence.

Add ModelJars and a qualified model marker

implementation("org.modeljars:modeljars:0.1.29")
implementation("org.modeljars.huggingface:ggml-org.qwen3-0.6b-gguf.q4_0:3.0.0-q4_0.1")

Backend	Java 25	Vector API	Native code
`backend-java`	Full pipeline	Numeric kernels	None
`backend-native`	Full pipeline	Remaining numeric kernels	Selected Rust bottlenecks via FFM

apple on-device inference

Use Apple Intelligence from the JVM

backend-apple connects Java 25 to Apple's SystemLanguageModel with FFM and a small Models-owned Swift binary. Apple manages the weights and model updates; your application checks availability and sends prompts without a model download or local server. The client also works directly with the Models LangChain4j and Spring AI adapters.

Supported Apple Silicon Mac
Apple Intelligence enabled and model ready
Java launched with native access enabled

Apple backend guide

// Add the Apple backend

implementation("com.integrallis:backend-apple:0.3.46")

// Check availability, then generate locally

try (var client = AppleFoundationModels.create()) {
    var status = client.availability();
    if (!status.available()) {
        throw new IllegalStateException(status.reason());
    }

    var response = client.generate(
        AppleFoundationModelsRequest.builder(
                "Summarize this document.")
            .instructions("Use one sentence.")
            .maxOutputTokens(64)
            .build());
}

integrations

Application-facing Java APIs

The same backend and diagnostics contracts serve plain Java, LangChain4j, Spring AI, Spring Boot, and guarded RAG. Spring AI ChatClient requests execute registered Java tools and return the follow-up model answer on blocking and streaming paths.

Qualified GGUF runtime

BERT/MiniLM encoders plus Llama, Qwen2, Qwen3, dense Qwen3.5, and Gemma 4 decoder families with memory-mapped quantized tensor paths.

Framework adapters

Blocking and streaming APIs for plain Java, LangChain4j, Spring AI, and Spring Boot applications.

Adaptive model routing

Explainable selection and safe failover across in-process and hosted clients using cost, quality, latency, live load, reliability, and conversation continuity.

In-JVM embeddings

Text to vectors from causal decoders and dedicated bidirectional encoders, tested against pinned llama.cpp references.

Streaming speech

Generate normalized PCM and WAV audio in process, with Soprano streaming its first playable chunk before the utterance completes.

Tool calling

Qualified Qwen, Hermes, Llama 3, Needle 2, Gemma 4, and MiniCPM5 call formats behind one contract, with Spring AI ChatClient callback execution, typed Needle result rendering, clean no-action responses, and schema-constrained decoding.

Guarded RAG

Retrieval abstention, trusted citations, unsupported-claim detection, and deterministic extractive fallback.

Exact qualification

Evidence binds model SHA, quantization, backend, host, correctness, TTFT, decode throughput, and end-to-end latency.

ModelJars metadata

Versioned upstream revisions, checksums, capabilities, domains, runtime requirements, and measured backend recommendations.

Local when required

The Models runtime stays in process, and the router's privacy-strict policy excludes hosted clients when prompts must not leave the JVM.

Browse qualified models Route local and hosted models