CafeAI：面向 Java 的可组合生成式 AI 框架发布 0.6.0

A foundational and composable framework for Gen AI in Java.

Try it in ten seconds — no clone, no build file, no project. With JBang:

jbang hello@akilisha/cafeai      # a web server in one file
jbang ask@akilisha/cafeai        # an LLM behind POST /ask (Anthropic, OpenAI, or local Ollama)
jbang init -t cafeai@akilisha/cafeai app.java   # start your own

The scripts are jbang/hello.java and jbang/ask.java, each one file.

CafeAI is not an invention of anything new. It is a deliberate re-orientation of familiar, battle-tested patterns and paradigms — Java's robustness, Express's composability, Langchain's AI primitives — unified into a foundational and composable framework for the AI age. Built for Java developers who refuse to trade understanding for convenience.

Why CafeAI?

The Java ecosystem deserves a serious Gen AI story. Not one hidden behind Spring Boot abstractions, but one built on first principles — where every layer is explainable, every concern is composable, and every design decision has a reason you can articulate in a conference talk.

The Name

Cafe → instantly recognizable as a coffee shop — Java
AI → the technology we're introducing
CafeAI → sounds like "kaf-ai" — a natural coming together of Java and AI

The Three Lineages

CafeAI stands deliberately on the shoulders of three proven traditions:

Installation

On Maven Central under com.akilisha.oss. CafeAI is modular — start with cafeai-core and add capability modules as you climb the adoption ladder.

Gradle

repositories { mavenCentral() }

dependencies {
    implementation 'com.akilisha.oss:cafeai-core:0.6.0'

    // add only what you use:
    implementation 'com.akilisha.oss:cafeai-config:0.6.0'         // application.properties/.yaml + profiles
    implementation 'com.akilisha.oss:cafeai-aiservices:0.6.0'         // app.agent() — LangChain4j AiServices
    implementation 'com.akilisha.oss:cafeai-agentic:0.6.0'        // CafeAgentic — LangChain4j langchain4j-agentic workflows
    implementation 'com.akilisha.oss:cafeai-memory:0.6.0'         // tiered context memory
    implementation 'com.akilisha.oss:cafeai-rag:0.6.0'            // retrieval-augmented generation
    implementation 'com.akilisha.oss:cafeai-guardrails:0.6.0'     // PII, jailbreak, toxicity, regulatory
    implementation 'com.akilisha.oss:cafeai-observability:0.6.0'  // OpenTelemetry tracing
    implementation 'com.akilisha.oss:cafeai-security:0.6.0'       // audit events for blocked prompt injection
    implementation 'com.akilisha.oss:cafeai-connect:0.6.0'        // Redis, Ollama, pgvector
    implementation 'com.akilisha.oss:cafeai-mustache:0.6.0'       // Mustache view engine
    implementation 'com.akilisha.oss:cafeai-session:0.6.0'        // SQLite-backed HTTP session store
    implementation 'com.akilisha.oss:cafeai-flight:0.6.0'         // JVM visibility via Flight Recorder -> OTel
    implementation 'com.akilisha.oss:cafeai-identity:0.6.0'       // verified callers (OIDC/OAuth); model calls without API keys
    implementation 'com.akilisha.oss:cafeai-mcp:0.6.0'            // app.mcp(): routes as tools for AI agents
    implementation 'com.akilisha.oss:cafeai-sentinel:0.6.0'       // AI Kubernetes/OpenShift incident pipeline
    testImplementation 'com.akilisha.oss:cafeai-test:0.6.0'       // record/replay model calls, evals
}

Maven

<dependency>
  <groupId>com.akilisha.oss</groupId>
  <artifactId>cafeai-core</artifactId>
  <version>0.6.0</version>
</dependency>

Using several modules? Import the BOM once and leave the versions off, so the modules cannot drift apart (they are released together and share service-loader interfaces, so a mix of versions is not a supported combination):

dependencies {
    implementation platform('com.akilisha.oss:cafeai-bom:0.6.0')
    implementation 'com.akilisha.oss:cafeai-core'
    implementation 'com.akilisha.oss:cafeai-guardrails'
    implementation 'com.akilisha.oss:cafeai-aiservices'
}

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>com.akilisha.oss</groupId>
      <artifactId>cafeai-bom</artifactId>
      <version>0.6.0</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

The BOM lists only CafeAI's own modules; LangChain4j and Helidon keep their own BOMs, which the modules import.

Requires Java 23+ (CI builds and tests on Java 23). For a local Jlama model, also add --add-modules jdk.incubator.vector --enable-native-access=ALL-UNNAMED to your run args.

Quick Start

var app = CafeAI.create();

// Infrastructure
app.ai(OpenAI.of("gpt-4o"));
app.memory(MemoryStrategy.mapped());      // SSD-backed via Java FFM — no Redis needed
app.observe(ObserveStrategy.otel());

// Safety
app.guard(GuardRail.pii());
app.guard(GuardRail.jailbreak());

// Persona
app.system("You are a helpful customer service agent for Acme Corp...");

// Routes
app.use(Middleware.json());
app.use(Middleware.rateLimit(60));

app.get("/health", (req, res, next) -> res.json(Map.of("status", "ok")));

app.post("/chat", (req, res, next) -> {
    res.stream(app.prompt(req.body("message")));
});

app.listen(8080, () -> System.out.println("☕ CafeAI is brewing on :8080"));

Read that out loud. A Java developer who has never touched Gen AI understands every line. An Express developer who has never touched Java understands the structure. A Python LangChain developer recognizes the concepts. That's three audiences, zero confusion.

Talk to it with any HTTP client. /chat streams the answer back as Server-Sent Events:

curl -N -H 'Content-Type: application/json' -H 'Accept: text/event-stream' \
     -d '{"message": "What is your return policy?"}' \
     http://localhost:8080/chat

# data: Our
# data:  return
# data:  policy
# data:  allows
# ...
# data: [DONE]

The Middleware Pipeline

Everything in CafeAI is middleware. HTTP concerns, AI concerns, security, observability, guardrails — all composable, all testable, all replaceable. This is the lesson Express taught us. CafeAI carries it to the AI age.

Incoming Request
   ↓
[ auth / JWT ]                  ← standard HTTP middleware
   ↓
[ rate limiter ]                ← standard HTTP middleware
   ↓
[ PII scrubber ]                ← security middleware
   ↓
[ jailbreak detector ]          ← security middleware
   ↓
[ prompt injection guard ]      ← security middleware
   ↓
[ guardrails PRE ]              ← ethical / regulatory middleware
   ↓
[ token budget enforcer ]       ← cost middleware
   ↓
[ semantic cache lookup ]       ← memory middleware (clean, prompt-only answers only)
   ↓
[ RAG retrieval ]               ← rag middleware
   ↓
[ LLM call / model router ]     ← ai middleware
   ↓
[ guardrails POST ]             ← ethical / regulatory middleware
   ↓
[ observability / OTel trace ]  ← observe middleware
   ↓
[ memory write ]                ← memory middleware
   ↓
[ streaming response ]          ← streaming middleware (SSE / WebSocket)

Every hard problem in Gen AI is a middleware concern. You can teach any one of those layers in isolation, then snap it into the pipeline. The pipeline is the curriculum.

The API Vocabulary

CafeAI introduces a deliberate vocabulary for AI-native Java development. Every name is guessable before you look it up.

Express-Parity HTTP (pound-for-pound)

app.get(path, handler)       // GET route
app.post(path, handler)      // POST route
app.put(path, handler)       // PUT route
app.delete(path, handler)    // DELETE route
app.use(middleware)          // global middleware
app.use(path, middleware)    // path-scoped middleware
app.listen(port)             // start the server

AI Infrastructure

app.ai(OpenAI.of("gpt-4o"))                    // register LLM provider
app.ai(Anthropic.of("claude-sonnet-4-5"))        // swap providers freely
app.ai(Gemini.of("gemini-3.6-flash"))           // Google Gemini — not a built-in type, added via ChatModelAccess
app.ai(Nvidia.of("moonshotai/kimi-k3"))         // NVIDIA API catalog (build.nvidia.com), key from $NVIDIA_API_KEY
app.ai(Ollama.of("llama3.3"))                   // local model via Ollama, no data leaves your infra
app.ai(Jlama.of("tjake/Qwen2.5-0.5B-Instruct-JQ4"))                     // pure-Java local model — in-process, no server
                                          //   run with: --add-modules jdk.incubator.vector
                                          //             --enable-native-access=ALL-UNNAMED
app.ai(ModelRouter.smart()                // smart routing — cheap vs expensive
        .simple(OpenAI.of("gpt-4o-mini"))
        .complex(OpenAI.of("gpt-4o")))
app.system("You are...")                  // system prompt — the AI's persona
app.template("name", "{{variable}}")     // named prompt templates

Memory

app.memory(MemoryStrategy.inMemory())     // Rung 1: JVM HashMap — prototype
app.memory(MemoryStrategy.mapped())      // Rung 2: SSD-backed via Java FFM
app.memory(MemoryStrategy.redis(config)) // Rung 3: Redis — the escape valve
app.memory(MemoryStrategy.hybrid())      // Rung 4: warm SSD + cold Redis

Semantic cache

app.cache(SemanticCache.inMemory(EmbeddingProvider.local()).build());

Repeat questions are answered from a cache matched by meaning, skipping the model call. A cache shared between users is also an attack surface — coax the model into a bad answer, get it stored, and it is served to everyone whose question lands nearby — so the defences are built in, not optional: only clean, prompt-only answers are stored (nothing a guardrail flagged; nothing with a session or RAG); entries are isolated per model and system prompt; a hit needs high embedding similarity and high word overlap and similar length (so a popular question with instructions appended does not match); every hit is re-screened by the current guardrails; and entries expire. .noCache() opts a call out. It is heuristic, not a guarantee — see docs/adr/ADR-013-semantic-cache-and-poisoning-defences.md for the threat model and its limits.

RAG

app.vectordb(PgVector.connect(config))   // vector store
app.embed(EmbeddingProvider.local())     // embedding model (ONNX via FFM)
app.ingest(Source.pdf("handbook.pdf"))   // ingest knowledge
app.ingest(Source.url("https://..."))
app.ingest(Source.directory("docs/"))
app.rag(Retriever.semantic(5))           // attach retrieval pipeline
app.rag(Retriever.hybrid(5))             // dense + sparse retrieval

Guardrails

app.guard(GuardRail.pii())               // PII detection — pre and post LLM
app.guard(GuardRail.jailbreak())         // adversarial prompt detection
app.guard(GuardRail.promptInjection())   // injected instructions — in input AND in retrieved RAG documents
app.guard(GuardRail.secrets())           // API keys, tokens, private keys — in and out
app.guard(GuardRail.promptLeak(system))  // the model repeating its own system prompt
app.guard(GuardRail.toxicity())          // harmful content filtering
app.guard(GuardRail.moderation(          // a moderation MODEL (LangChain4j's), not a pattern list
    OpenAI.moderation("omni-moderation-latest")))
app.guard(GuardRail.regulatory()         // GDPR, HIPAA, FCRA, CCPA
    .gdpr().hipaa())
app.guard(GuardRail.topicBoundary()      // scope enforcement
    .allow("customer service", "orders")
    .deny("politics", "medical advice"))

Agents & Tools

// You define a typed interface; CafeAI binds it to a LangChain4j AiService —
// which owns the reasoning loop, tool dispatch, and chat memory — and gives it
// an HTTP identity: session threading, guardrail pre-screening, observability.
app.agent("support", SupportAgent.class)
    .system("You are a support specialist for Acme.")
    .tool(new OrderLookupTool())                   // @Tool-annotated Java methods
    .memory(MemoryStrategy.inMemory())
    .guard(GuardRail.jailbreak())                  // runs before the agent loop
    .configure(b -> b /* full AiServices.Builder escape hatch */);

CafeAI does not reimplement the ReAct loop, tool protocol, or MCP client — LangChain4j does all of that. CafeAI writes the HTTP binding. MCP servers attach through LangChain4j's MCP support; multi-agent workflows are a supervisor agent calling sub-agents as tools, or a middleware chain — not bespoke primitives.

Observability

app.observe(ObserveStrategy.otel())       // OpenTelemetry — production
app.observe(ObserveStrategy.console())    // console — development

Configuration

// declared once, at the point of use — a self-documenting, self-registering key
static final ConfigKey<Integer> CHUNK_SIZE =
    ConfigKey.of("cafeai.rag.chunk.size", Integer.class, 500, "Chunk size in characters");

int size = AppConfig.load().get(CHUNK_SIZE);

# application.yaml on the classpath — or application-{profile}.yaml, or an
# external file pointed to by CAFEAI_CONFIG_FILE / cafeai.config.file
cafeai:
  rag:
    chunk:
      size: 800

ConfigKey/AppConfig live in cafeai-core — every module can declare a tunable value with no new dependency. Add cafeai-config to your application to turn those keys into real, overridable settings — system property, environment variable, profile file, or application.yaml — resolved by Helidon Config, no CafeAI-invented naming scheme. Without it, every key just resolves to its own coded default, exactly as before this existed. See docs/adr/ADR-012-application-config.md.

cafeai-sentinel — AI Cluster Incident Pipeline

SentinelConfig config = SentinelConfig.create().namespace("payments");
ClusterWatch watch = new ClusterWatch(config);

app.agent("cluster-investigator", ClusterInvestigator.class)
    .model(Anthropic.of("claude-sonnet-4-5-20250929"))
    .tool(new KubeTools(watch.client(), "payments", Redactor.of(config.isRedact())));

IncidentTracker tracker = new IncidentTracker(config)
    .onIncident(IncidentSink.of(new LogSink(), new SsePublisher()))
    .investigator(incident -> /* run the agent above */ null)
    .start();

watch.onPodState(tracker::accept);
watch.start();

Watches one Kubernetes/OpenShift namespace, triages pod failures with rules (no model), coalesces correlated failures into one incident per broken workload, and runs a read-only agentic investigation on confirmed incidents — secrets and PII redacted before anything reaches the prompt. A pipeline, not a product: it ends at "incident published," fanned out to a log, a webhook, an SSE stream, or your own IncidentSink. See docs/roadmap/ROADMAP-18-sentinel.md and the runnable capstones/cluster-sentinel companion.

cafeai-identity — Callers, Not API Keys

var issuer = Issuer.discover("https://issuer.example.com/realms/acme");
app.filter(Auth.bearer(issuer, "orders-api"));                       // or Auth.login(...) for browsers
app.get("/reports", Auth.require(Auth.group("finance")), reports);

app.ai(OpenAI.of("<model-id>").withBaseUrl("https://models.internal.example.com/v1")
        .withCredentials(OAuthCredentials.tokenExchange(issuer, "orders-api", secret, "model-gateway")));

Static API keys are banned in most companies: nothing ties a call to a person, and no one person's access can be revoked. With cafeai-identity, every request has a verified caller from any OpenID Connect issuer, and that caller goes everywhere:

• Sign-in: API tokens, browser sign-in (tokens kept server-side), or terminal sign-in.
• Model calls: made on the caller's behalf (RFC 8693 token exchange), with no API key.
• Conversations and records: conversations belong to whoever started them, and usage and audit records name the caller.
• RAG: the database decides what each caller may read (PostgreSQL row-level security).
• MCP and WebSockets: both are protected the same way.

Built on open standards only, and tested against Keycloak. See DEVELOPER_GUIDE §32 and docs/design/IDENTITY.md.

Module Structure

cafeai/
├── cafeai-bom            ← Bill of materials: one version for every published module
├── cafeai-core           ← Express-style API, routing, middleware chain, all AI primitives
├── cafeai-config         ← File-based config (application.properties/.yaml + profiles) for AppConfig
├── cafeai-aiservices         ← Binds LangChain4j AiServices to the HTTP server — app.agent()
├── cafeai-agentic        ← Binds LangChain4j langchain4j-agentic multi-agent workflows — CafeAgentic
├── cafeai-memory         ← Tiered context memory (in-memory, FFM/SSD, Redis)
├── cafeai-rag            ← Document ingestion, chunking, embedding, retrieval, vector DBs
├── cafeai-guardrails     ← PII, jailbreak, toxicity, regulatory compliance
├── cafeai-observability  ← OpenTelemetry tracing, console logging
├── cafeai-security       ← Blocks prompt injection, raises audit events
├── cafeai-connect        ← Out-of-process services: Redis, Ollama, pgvector
├── cafeai-mustache       ← Optional Mustache view engine
├── cafeai-session        ← HTTP session store for Middleware.session() (SQLite, single-instance)
├── cafeai-flight         ← JVM-level visibility via Java Flight Recorder, surfaced as OTel metrics
├── cafeai-identity       ← Verified callers (OpenID Connect, OAuth 2.0) through the AI layer; no API keys
├── cafeai-mcp            ← MCP server: chosen routes and @Tool objects as tools for AI agents
├── cafeai-dev            ← Development only: save a file, the app reloads in about half a second
├── cafeai-sentinel       ← AI cluster incident pipeline for Kubernetes / OpenShift (ROADMAP-18)
├── cafeai-test           ← Tests only: record/replay model calls (no API key, no cost) and evals
└── cafeai-examples       ← Runnable reference implementations — the adoption ladder

Each module is an independent rung on the adoption ladder. Start with cafeai-core. Graduate when you're ready.

The Incremental Adoption Ladder

CafeAI is structured so that every team can start at the bottom and climb deliberately.

Modern Java Feature Map

CafeAI treats modern Java (21–23) features as load-bearing architecture — not demos.

Tiered Memory Architecture

Hot    →  JVM Heap           (active conversation turn)
Warm   →  FFM MemorySegment  (recent sessions — SSD-backed, no network)
Cold   →  Redis               (distributed — the escape valve)
Frozen →  Vector DB          (semantic long-term memory, RAG corpus)

The key insight: most applications do not need Redis. The SSD-backed FFM tier handles production single-node deployments with zero network overhead, zero cloud tax, and crash-recovery for free. Redis is the escape valve — not the default.

Technology Stack

Running the Examples

# Clone
git clone https://github.com/akilisha/cafeai.git
cd cafeai

# Run the hello world example
./gradlew :cafeai-examples:run

# Run a specific example
./gradlew :cafeai-examples:run -PmainClass=io.cafeai.examples.JlamaStreamingExample

# Smoke-test a real provider with your own key (skips itself without one)
./gradlew :cafeai-core:liveTest

Live tests, the keys they read and the model variables are described in GETTING-STARTED.md.

Capstones

Full reference applications live in capstones/ — support-desk, meridian-qualify, acme-claims, invoice-processor. They build against the framework as project(':cafeai-*'), so they break the build if an API changes. Not published.

./gradlew :capstones:invoice-processor:run -Pdry

Blog Series

A narrative arc through the framework, one capability at a time, each post anchored to working code — not one post per module.

1. Brewing AI in Java — Introducing CafeAI
2. The Middleware Pattern Meets Gen AI — From Express to CafeAI
3. Your First LLM Call Without Spring Boot — Helidon SE + LangChain4j
4. Prompt Engineering in Java — Templates, System Prompts, and the API Vocabulary
5. Context Memory Without the Cloud Tax — Java FFM and the Tiered Memory Model
6. Building a RAG Pipeline in Java — Ingestion, Embedding, and Retrieval
7. Tool Use in Java — Giving the LLM Actions to Take
8. Ethical Guardrails as Middleware — PII, Jailbreak, and Regulatory Compliance
9. Vision and Audio in Java — Multimodal AI Without the Boilerplate
10. Structured Output — Typed LLM Responses, No Parser Required
11. Production-Grade AI — Token Budgets, Retries, Observability, and Incident Response
12. The Capstone Series — Five Applications and What Each Found

License

Apache 2.0

CafeAI: Not an invention of anything new — a re-orientation of everything proven.