For a decade the pattern for running a language model next to a Java service was fixed: the JVM handled the application, and a Python process — spun up as a sidecar, called over HTTP, or wrapped behind a serving framework — handled inference. That split existed for a real reason: PyTorch and NumPy call straight into native BLAS and CUDA, and Java had no comparable way to move numeric loops onto real SIMD hardware without JNI.
That reason is quietly going away. Two open-source projects — llama3.java by Alfonso² Peterssen and jlama by T Jake Luciani — now run transformer inference entirely inside the JVM: no JNI, no Python, no ONNX Runtime, no native .so file. This article explains the two JDK features that made that possible, then builds a working SIMD kernel that shows the actual trick.
Why this is news
Two JDK APIs converged to make in-process, native-code-free inference viable:
The Foreign Function & Memory API (JEP 454, finalized in JDK 22) standardized java.lang.foreign.MemorySegment and Arena — off-heap, memory-mapped byte access that never touches the garbage collector. That’s what makes it practical to mmap a multi-gigabyte GGUF weight file straight into a Java process, without copying gigabytes onto the heap and without writing a line of JNI to do it.
The Vector API (jdk.incubator.vector, incubating since JDK 16 through the JDK 21–24 cycle, still gated behind --add-modules as of this writing) gives Java explicit SIMD types — FloatVector, IntVector, ByteVector — that compile down to real AVX2, AVX-512, or NEON instructions on the running CPU, instead of hoping HotSpot’s auto-vectorizer notices a hot loop and does the right thing. That’s the piece that turns a plain for loop dot product into something running at a meaningful fraction of native BLAS speed.
Neither API was built for LLM inference specifically. Both happened to be exactly what it needed.
Two real projects, two different scopes
llama3.java is deliberately minimal: a small, dependency-free Java implementation of Llama 3 and Mistral inference, in the spirit of Andrej Karpathy’s llama2.c but targeting the JVM. It memory-maps GGUF files, supports the common quantized formats (Q8_0, Q4_0), and leans on the Vector API for the matmul hot loop. There’s nothing to install beyond a recent JDK — it’s meant to be read end to end in an afternoon.
jlama is the production-shaped version of the same idea: it supports multiple model families (Llama, Mistral, Gemma, and BERT-style embedding models), loads both GGUF and SafeTensors checkpoints, implements several quantization schemes, and ships integrations for LangChain4j and an OpenAI-compatible REST server, so it can drop into an existing Java service as a library rather than a standalone demo.
Different scopes, same two load-bearing pieces underneath: memory-mapped off-heap weights, and a Vector API kernel doing the arithmetic.
How it actually maps onto Java
A transformer’s forward pass is dominated by one operation, repeated constantly: multiply a weight matrix by an activation vector. Concretely, that’s a dot product per output neuron, done for every linear layer, for every one of the model’s layers, for every token generated. Everything else — layer norm, softmax, positional encoding — is comparatively cheap. So the entire performance problem reduces to: how fast can this JVM compute a dot product, and how fast can it get the weights into the CPU in the first place?
Weights come from an Arena-backed, memory-mapped MemorySegment over the model file — the OS page cache does the actual I/O, and the JVM heap never sees the multi-gigabyte weight tensors, so there’s no GC pressure from loading a model. The dot product runs through the Vector API. And because token-by-token decoding needs each previous token’s key/value projections again on every step, a KV cache keeps those around across decode steps instead of recomputing them — the one piece of “state” pure-Java inference has to manage explicitly.
flowchart LR
FILE["GGUF weight file<br/>(disk)"] -- "FileChannel.map<br/>READ_ONLY" --> SEG["Off-heap MemorySegment<br/>(zero-copy, no GC)"]
SEG --> KERNEL["SIMD matmul kernel<br/>jdk.incubator.vector.FloatVector"]
TOK["Input tokens"] --> EMB["Embedding lookup"]
EMB --> BLOCK["Transformer block<br/>(attention + feed-forward)"]
KERNEL --> BLOCK
BLOCK -- "repeat per layer" --> BLOCK
BLOCK --> KV["KV cache<br/>(reused across decode steps)"]
KV --> BLOCK
BLOCK --> SAMPLE["Sample next token"]
SAMPLE -- "feed back" --> TOK
That whole loop runs inside one JVM process. Compare it to the architecture it replaces:
flowchart TB
subgraph SIDE ["Classic: JVM app + Python sidecar"]
APP1["Java service"] -- "HTTP / gRPC" --> PY["Python process<br/>(PyTorch + native BLAS/CUDA)"]
PY --> GPU1["GPU or CPU"]
end
subgraph PURE ["Pure-Java: single JVM"]
APP2["Java service"] --> ENGINE["jlama / llama3.java<br/>(in-process)"]
ENGINE -- "jdk.incubator.vector" --> CPU2["CPU SIMD"]
end
No second language, no second process to deploy and monitor, no serialization across a process boundary for every request.
A working SIMD kernel
This is the actual trick, stripped down to the operation that matters. SPECIES_PREFERRED asks the JVM for the widest vector the running CPU supports — 8 floats on AVX2, 16 on AVX-512, 4 on NEON — so the same compiled bytecode adapts to whatever hardware it lands on, without shipping separate native builds per platform:
import jdk.incubator.vector.FloatVector;
import jdk.incubator.vector.VectorOperators;
import jdk.incubator.vector.VectorSpecies;
public class SimdDot {
private static final VectorSpecies<Float> SPECIES = FloatVector.SPECIES_PREFERRED;
// One row of a weight matrix, dotted with the activation vector.
// Repeated per output neuron, this IS a transformer's linear layer.
static float dot(float[] weightsRow, float[] activations) {
int width = SPECIES.length();
int bound = SPECIES.loopBound(weightsRow.length);
FloatVector acc = FloatVector.zero(SPECIES);
int i = 0;
for (; i < bound; i += width) {
FloatVector w = FloatVector.fromArray(SPECIES, weightsRow, i);
FloatVector a = FloatVector.fromArray(SPECIES, activations, i);
acc = w.fma(a, acc); // fused multiply-add, one CPU instruction where available
}
float sum = acc.reduceLanes(VectorOperators.ADD);
for (; i < weightsRow.length; i++) { // scalar tail for the remainder
sum += weightsRow[i] * activations[i];
}
return sum;
}
}
Run it with --add-modules jdk.incubator.vector on the classpath and it compiles and executes as ordinary Java — no compiler flags for native code, no build-time codegen, no platform-specific artifact.
Quantization is the other half of the story, and it’s why jlama and llama3.java both store weights as Q4_0 or Q8_0 rather than raw floats: each weight becomes a small integer plus a per-group scale factor, dequantized on the fly. A quantize-aware dot product conceptually just widens each integer lane back to float before the same fused-multiply-add:
// weightsQ8: quantized int8 weights, one scale per group of the row
static float dequantizeThenAccumulate(byte weightQ8, float activation, float scale) {
return (weightQ8 * scale) * activation;
}
The real, fully vectorized version — widening ByteVector lanes into IntVector or FloatVector lanes before the multiply — is where jlama’s and llama3.java’s actual kernels get intricate, and it’s worth reading their source directly if you want the complete trick. The reason it’s worth the complexity: it isn’t really about saving CPU cycles.
Where the time actually goes
At batch size one — one user, one conversation, generating one token at a time — transformer decoding is memory-bandwidth bound, not compute bound. Every generated token requires streaming the entire weight set past the CPU once. Throughput is governed by memory bandwidth divided by model size in bytes, not by how many FLOPs the CPU can theoretically issue. That’s precisely why quantization is the highest-leverage lever available: storing a weight as one byte instead of four cuts the bytes that have to move, directly and proportionally.
This also sets honest expectations. Pure-Java CPU inference lands in roughly the same range as llama.cpp’s CPU path, because both are bound by the same wall — a handful to a few dozen tokens/second for a 7–8B model on a modern multi-core desktop, depending on quantization and core count. GPU-backed Python serving stacks are still meaningfully faster, because GPU memory bandwidth is far higher and batching amortizes it across many concurrent requests. Pure-Java inference isn’t trying to win that race. It’s trading peak throughput for collapsing a two-language, two-process deployment into one JVM.
When this makes sense
It’s a good fit when the deployable itself is the point: a single JVM artifact, inference that never leaves the process (useful when the data genuinely can’t leave it either), and a team that would rather lean on JVM tooling it already runs — heap and GC monitoring, virtual threads for the request-handling layer — than stand up and operate a second runtime.
It’s the wrong tool for maximum-throughput, high-concurrency serving at real scale — that’s GPU-plus-batching territory, a different problem with a different answer. And if what you actually want is native-speed bridging rather than pure Java, the Foreign Function & Memory API opens a more direct path to calling llama.cpp itself; if you want this wrapped in an existing enterprise stack, LangChain4j’s Quarkus integration embeds a local model behind a familiar service layer. Both are worth their own look — this article is about the pure-Java path specifically.
Production checklist
- The Vector API is still an incubator module — run with
--add-modules jdk.incubator.vector, and pin the exact JDK version, since incubator APIs can shift between releases - Load weights through
Arena.ofShared()+FileChannel.map(MapMode.READ_ONLY, …)— never read multi-gigabyte weights into on-heap arrays; off-heap segments skip the GC entirely - Budget for JIT warm-up — the first few forward passes are slower until C2 compiles the vectorized loops; a warm-up inference at startup keeps that cost off the first real user
- Size the inference thread pool to physical cores, not vCPU count — SIMD kernels are compute-bound per call, so oversubscribing just adds context-switch overhead; keep virtual threads for request handling, not for the numeric loop itself
- Quantize (Q4/Q8) before optimizing anything else — it’s the single biggest lever on a memory-bandwidth-bound workload
- If targeting GraalVM native-image, test it explicitly and early — incubator modules and vector intrinsics don’t always survive native-image analysis cleanly
Closing thoughts
None of this required a new machine-learning breakthrough. It required two general-purpose JDK APIs — one for touching memory without the GC, one for touching the CPU’s SIMD units without JNI — and two projects willing to use them for something nobody had bothered to build in Java before. The result is a genuinely different deployment shape: an LLM as a JVM library instead of a network dependency. For a Java shop that already trusts its own runtime more than it trusts a second one, that shape is worth knowing about.
Further reading: