In-process inference - Java 25

Models

Run open-weight language models inside your Java application, without Python, a separate model server, or a network request.

Models loads GGUF files and owns tokenization, the transformer graph, KV state, sampling, and streaming. Both backends use the Java 25 runtime and Vector API. backend-java executes every kernel in Java; backend-native substitutes only selected measured bottleneck kernels with Models-owned Rust through FFM.

// Generated reference to a qualified artifact

import static org.modeljars.catalog.Qwen3_0_6b_Q4_0.MODEL;

try (var model = ModelJars.open(MODEL)) {
    String answer = model.generate(prompt, options);
}

// Exact Maven-style marker coordinate

try (var model = ModelJars.open(
        "org.modeljars.huggingface:" +
        "ggml-org.qwen3-0.6b-gguf.q4_0:" +
        "3.0.0-q4_0.1")) {
    String answer = model.generate(prompt, options);
}

// Apple's on-device SystemLanguageModel

try (var client = AppleFoundationModels.create()) {
    if (client.availability().available()) {
        String answer = client.generate(prompt).text();
    }
}

Read the documentation View source

runtime

Run models directly inside your Java application

A GGUF file packages the model structure, tokenizer metadata, and trained weights. Models maps those weights, executes the transformer, maintains its attention cache, and generates tokens in process. Backend selection changes only kernels justified by exact artifact and host evidence.

Add ModelJars and a qualified model marker

implementation("org.modeljars:modeljars:0.1.0")
implementation("org.modeljars.huggingface:ggml-org.qwen3-0.6b-gguf.q4_0:3.0.0-q4_0.1")

Backend	Java 25	Vector API	Native code
`backend-java`	Full pipeline	Numeric kernels	None
`backend-native`	Full pipeline	Remaining numeric kernels	Selected Rust bottlenecks via FFM

apple on-device inference

Use Apple Intelligence from the JVM

backend-apple connects Java 25 to Apple's SystemLanguageModel with FFM and a small Models-owned Swift binary. Apple manages the weights and model updates; your application checks availability and sends prompts without a model download or local server. The client also works directly with the Models LangChain4j and Spring AI adapters.

Supported Apple Silicon Mac
Apple Intelligence enabled and model ready
Java launched with native access enabled

Apple backend guide

// Add the Apple backend

implementation("com.integrallis:backend-apple:0.1.0")

// Check availability, then generate locally

try (var client = AppleFoundationModels.create()) {
    var status = client.availability();
    if (!status.available()) {
        throw new IllegalStateException(status.reason());
    }

    var response = client.generate(
        AppleFoundationModelsRequest.builder(
                "Summarize this document.")
            .instructions("Use one sentence.")
            .maxOutputTokens(64)
            .build());
}

integrations

Application-facing Java APIs

The same backend and diagnostics contracts serve plain Java, LangChain4j, Spring AI, Spring Boot, and guarded RAG.

Qualified GGUF runtime

Llama, Qwen2, and Qwen3 decoder families with F32, F16, Q4_0, Q5_0, Q8_0, Q4_K, Q5_K, and Q6_K tensor paths.

Framework adapters

Blocking and streaming APIs for plain Java, LangChain4j, Spring AI, and Spring Boot applications.

Guarded RAG

Retrieval abstention, trusted citations, unsupported-claim detection, and deterministic extractive fallback.

Exact qualification

Evidence binds model SHA, quantization, backend, host, correctness, TTFT, decode throughput, and end-to-end latency.

ModelJars metadata

Versioned upstream revisions, checksums, capabilities, domains, runtime requirements, and measured backend recommendations.

Private by construction

Prompts, retrieved context, model weights, and generated text remain inside the application process.

Browse qualified models Run the notebooks