Beyond Retrieval: The CAG Blueprint for LLM Memory

Retrieval-Augmented Generation (RAG) solved an important problem: it gave language models a way to work with knowledge that was too large, too private, or too dynamic to live in the model itself. But retrieval also inserted a new runtime dependency into every request. Documents must be chunked, indexed, searched, re-ranked, assembled, and then passed back to the model before reasoning can begin.

For many workloads, that trade is correct. For others—especially structured, repetitive, latency-sensitive workloads—the retrieval layer can become the system's dominant source of delay and context fragmentation.

Cache-Augmented Generation (CAG) changes the question from “What should I retrieve now?” to “What stable knowledge can I prepare once and reuse?” The architecture precomputes attention state for a stable knowledge base, keeps that state available as a KV cache, and lets new queries attend directly over the cached representation. The result is not a universal replacement for RAG. It is a different memory tier, and it becomes most useful when paired with retrieval rather than treated as an all-or-nothing alternative.

1. Why Runtime Retrieval Hits a Ceiling

A conventional RAG pipeline has several moving parts between a user query and the model: chunking, a vector database, runtime search, and often a re-ranking stage. Every layer can add useful filtering, but every layer is also another place where latency or selection error can enter the request path.

The slide frames the failure surface in three categories:

• Retrieval latency. Real-time vector search introduces work that must complete before generation.
• Token-level noise. Chunking can detach facts from the relationships that give them meaning.
• Selection errors. Similarity search can omit a bridging document that is essential for multi-hop reasoning.

The presentation illustrates those concerns with scenario-specific figures—80ms+ retrieval latency, 40% context loss, and a 50% multi-hop failure rate. Those numbers should be read as claims from the supplied material rather than universal properties of RAG. The architectural point is broader: once retrieval is on the critical path, its latency and recall become part of the model's effective reasoning envelope.

For unstructured corpora, that may be acceptable. For highly structured inputs, a missed relationship can be worse than a missed paragraph.

2. The Schema-Linking Problem Is a Structural Problem

Structured data exposes a weakness in chunk-centric retrieval. A schema is not a bag of independent facts. Foreign keys, constraints, aliases, and table relationships define a graph. If those relationships are split across embeddings and retrieved independently, the model can receive all the right nouns while still missing the logic that connects them.

The supplied slide contrasts a fragmented RAG view of a relational schema with a CAG view in which the full structure remains connected. It also cites a “GWTG-HF Registry Study” breakdown of errors: 60.9% logical reasoning errors, 21.7% retrieval failures, and 17.4% SQL syntax errors. The source material does not provide the underlying study methodology, so the percentages are best treated as presentation-level evidence for the stated failure modes rather than independently verified benchmarks.

The engineering lesson is still useful: retrieval recall and relational integrity are different objectives. A search system can retrieve semantically similar chunks and still fail to preserve join paths, constraints, or dependency order. This matters in SQL generation, API orchestration, program analysis, configuration reasoning, and automated testing—anywhere the meaning of one artifact depends on another.

3. Enter Cache-Augmented Generation

CAG moves stable context out of the runtime retrieval loop. Instead of decomposing the knowledge base and fetching a subset for every query, it loads the entire eligible knowledge base during an offline phase and stores the resulting model attention state.

The contrast is architectural:

RAG

1. Build or update chunks.
2. Store searchable representations in a vector database.
3. Retrieve a subset at request time.
4. Assemble the selected context.
5. Run the LLM.

CAG

1. Load the stable knowledge base.
2. Run an offline prefill.
3. Persist the resulting transient model state as a KV cache.
4. At runtime, submit only the new query against that precomputed state.

This eliminates the retrieval stage from the hot path. It does not make inference literally free or remove every possible reasoning failure. What it does remove is runtime document selection for the cached tier. If the required fact was included in the preloaded knowledge base, it cannot be omitted because a similarity search failed to surface it.

That distinction—selection avoidance rather than better selection—is the central architectural move.

4. The Computational Physics of KV Caching

The reason CAG can change Time-to-First-Token (TTFT) is that transformer inference already exposes a reusable intermediate representation: the key and value tensors produced by self-attention.

The slide expresses the offline step as:

C_KV = KV-Encode(D)

where D is the background knowledge and C_KV is the cached key/value state. At runtime, the model computes attention for the new query while referencing that cache:

A = M(Q | C_KV)

The practical optimization is straightforward: pay the prefill cost for stable context once instead of paying it on every request.

The supplied HotPotQA-Large annotation reports generation time moving from 92.08s for an in-context path to 2.26s for CAG, labeled as a 40x speedup. As with the other benchmark figures in the deck, this is workload-specific rather than a general latency guarantee.

The deeper point is independent of the exact number. Repeatedly re-reading the same stable prefix is computational duplication. If the model, cache format, and knowledge prefix are compatible, precomputing those attention states converts repeated prefill work into a reusable memory artifact.

5. Preserving Structure Changes Query Fidelity

A cache can preserve the full arrangement of structured context in a way that chunk retrieval may not. That matters most when the query needs multiple related fields, constraints, or cross-document dependencies.

The comparison slide reports:

• One-field query accuracy: 88.0% for standard RAG and 94.5% for optimized RAG; CAG is labeled “Near-Perfect.”
• Three-field query accuracy: 10.0% for standard RAG and 82.0% for optimized RAG; CAG is labeled “High Fidelity.”
• Relational integrity: low for standard RAG, moderate for optimized RAG, and high for CAG because the DDL remains intact in memory.
• Hallucination vulnerability: highest where required metadata is missing, lower as the model is grounded in unfragmented context.

The article should not turn those labels into universal accuracy claims. Their value is as a diagnostic model: the more a task depends on relationships rather than isolated facts, the more expensive context fragmentation becomes.

This suggests a useful evaluation strategy. Do not test only single-fact lookup. Include multi-table joins, cross-file dependencies, multi-hop constraints, and queries that are impossible to answer correctly unless the system preserves the connective tissue.

6. CAG as a Test-System Memory Layer

The case-study material applies the architecture to zero-shot regression testing. The depicted system, “Cleverest,” forms a feedback loop from a commit diff to a prompt synthesizer, through an LLM module using CAG, into a sandboxed interpreter, and then back from error output into the next iteration.

The important architectural idea is not the product name; it is the memory boundary. The LLM can repeatedly process code changes while retaining a stable interpreter rulebook or structured environment in its cached context. That reduces the chance that each iteration sees a slightly different subset of the system's grammar, schema, or testing constraints.

The slide reports that the system found bugs in under three minutes for highly structured formats such as JavaScript and XML. It also calls out a limitation: highly compact, non-human-readable binary formats such as PDF can cause early exits when strict constraints dominate the generation process. Those are case-study claims from the supplied material, not independently verified performance results.

The presence of a sandboxed interpreter is also an important production signal. Generation and execution should remain separate trust domains. A cached context can improve consistency, but it does not make generated code safe to execute without isolation, limits, and observable failure handling.

7. Hybrid Fuzzing: Use CAG to Generate Better Seeds

The next slide extends the testing loop into fuzzing. Instead of expecting an LLM to replace a fuzzer, CAG is used to produce semantically informed seeds that can be fed into a conventional greybox fuzzing engine.

The supplied comparison labels WAFLGo at roughly 13–15 hours and ClevFuzz at roughly 6 hours, arguing that CAG-generated test cases reduce the time needed to reach useful mutations. The slide also claims the CAG seed gets the fuzzer “90% of the way to the vulnerability instantly.” That language is specific to the presentation and should not be generalized beyond the described case.

Architecturally, the pattern is strong: use the LLM where semantic structure matters, and use deterministic search where coverage matters.

A production testing stack can therefore divide responsibilities:

• CAG maintains stable grammar, schema, and system rules.
• The LLM synthesizes high-quality initial inputs.
• A fuzzer explores the mutation space at machine speed.
• A sandbox executes candidates and returns structured failures.
• The loop feeds new evidence back into the next generation step.

That is a more robust role for an LLM than asking it to be both the semantic planner and the exhaustive search engine.

8. Prompt Caching Changes the Cost Model

CAG is not only about latency. When a provider offers discounted cache reads, repeated requests against a stable prefix can change the economics of a high-volume workflow.

The scenario in the slide assumes a 40k-token stable base, a 2k-token query, 1,000 requests per day, and a pricing model labeled “Claude Sonnet 4.6.” Under the slide's assumptions, continuously prefilling the entire 42k tokens costs $120/day, while caching the 40k stable base and paying full price only for new query tokens produces a stated total of $12.15/day.

The important takeaway is not the exact dollar amount—provider prices and cache policies change. It is the cost equation:

Stable tokens should not be billed as if they were novel on every turn when the platform can reuse them.

This makes workload shape a first-class architectural input. Caching becomes more attractive when:

• the stable prefix is large,
• the number of repeated requests is high,
• the delta per request is small,
• cache-hit rates are predictable,
• and cache retention aligns with traffic cadence.

A correct cost model should include misses, rebuilds, TTL expiry, storage charges where applicable, and version changes to the stable base.

9. Provider Cache Semantics Are Part of the Architecture

The provider comparison slide shows that “prompt caching” is not one identical feature across APIs.

In the supplied snapshot:

• Anthropic (Claude) is shown with explicit API control, a 90% cache-read reduction, a minimum scale of 2,048 tokens, and a 5-minute default TTL with keepalive requirements for sporadic traffic.
• Google (Gemini 2.5) is shown with implicit and explicit caching, a 90% read reduction, a 1,024-token minimum, and a 1-hour default TTL plus storage fees for explicit caches.
• OpenAI is shown with automatic prefix matching, a 50% read reduction, a 1,024-token minimum, and less explicit developer control.

These values come from the supplied slide and should be treated as a provider-policy snapshot, not a durable contract. Production code should read current provider documentation before making cost or retention guarantees.

More importantly, cache semantics leak into application design. TTL determines whether sporadic workloads retain value. Prefix-matching rules influence prompt layout. Explicit cache identifiers can simplify versioning. Eviction behavior can turn a theoretically cheap architecture into an expensive one if the workload does not sustain cache residency.

The API is not just transport. It defines the memory model you can actually operate.

10. The VRAM Wall

CAG trades retrieval work for memory residency. That means its scaling limit is not only the model's nominal context window—it is also the hardware footprint of the KV state.

The hardware slide estimates 0.5MB–2.5MB per token and states that an 8,000-token context can consume roughly 8GB of VRAM per request on an 8B model. Those figures are implementation-dependent, but they illustrate the key constraint: cached attention grows with the amount of context retained.

The deck names three optimization directions:

• CoinRAG & TurboRAG: compress context into semantic nuggets and move cross-chunk attention work offline.
• CacheBlend: selectively recompute subsets of tokens to balance speed with contextual accuracy.
• State-Space Models (SSMs): maintain a fixed-size internal state rather than scaling transformer KV memory linearly with sequence length.

This turns memory planning into a capacity-engineering problem. A CAG deployment needs to model concurrent sessions, cache replication, model parallelism, eviction policy, and the worst-case working set. A fast cache that forces aggressive eviction under real concurrency is not a fast system.

11. Why Pure CAG Is Not a Universal Answer

The strongest slide in the deck may be the one that argues against pure CAG.

It identifies three structural limits:

1. Boundary of scale. Large codebases and enterprise datasets can exceed practical context limits and physical memory budgets.
2. High volatility. Transactional or rapidly changing data can make cached state stale, forcing frequent rebuilds.
3. Attention dilution. Very large preloads can trigger “lost-in-the-middle” behavior, where important details buried deep in the context receive insufficient attention.

These constraints reveal the correct abstraction: CAG is a memory tier for stable, high-utility knowledge, not an infinite database replacement.

A production cache therefore needs a lifecycle. Stable content should be versioned, warmed, observed, and invalidated. Volatile data should avoid the cache or enter through a separate path. When a schema changes, the system should be able to identify which cache artifacts are stale and rebuild them without taking the entire service offline.

12. The Tiered Hybrid Topology

The synthesis slide proposes a two-tier memory architecture.

Tier 0 — CAG cache contains stable, high-utility context: database schemas, API documentation, standard libraries, system prompts, and other definitions that change slowly. The goal is structural integrity and low-latency reuse.

Tier 1 — RAG index contains dynamic, long-tail context: active code diffs, recent logs, runtime errors, execution feedback, and other information whose freshness matters more than cache reuse.

Above both tiers sits the LLM self-attention layer. The model reasons over a stable foundation while retrieval injects only the volatile delta.

This topology is stronger than a simple “CAG versus RAG” choice because it partitions knowledge by rate of change. Stability determines what belongs in cache; freshness determines what belongs in retrieval.

That partition also creates an operational contract:

• Tier 0 needs versioning, warm-up, residency, and cache-hit observability.
• Tier 1 needs indexing freshness, retrieval quality, re-ranking quality, and source attribution.
• The assembly layer needs deterministic precedence rules when a dynamic fact overrides a cached definition.
• Evaluation must test both cache correctness and retrieval freshness.

13. A Decision Taxonomy for CAG, RAG, and Hybrid Systems

The decision tree in the presentation starts with two questions: how large is the knowledge base, and how quickly does it change?

The slide's rule of thumb is:

• If the context fits within roughly 500k tokens and schema drift is low, deploy CAG.
• If the context fits but includes highly volatile transactions or diffs, deploy a hybrid with stable definitions in Tier 0 and dynamic updates in Tier 1.
• If the system exceeds the context boundary and exhibits high drift, deploy RAG for scale and freshness.

The 500k threshold is a heuristic from the supplied material, not a universal platform boundary. The more durable decision criteria are these:

Choose CAG when the knowledge is bounded, repetitive, structurally sensitive, and slow-changing.

Choose RAG when the knowledge is very large, rapidly changing, sparse per query, or impractical to keep resident.

Choose hybrid when the application contains both kinds of knowledge—which is the common case in enterprise systems.

14. The Blueprint for LLM Memory

The final blueprint condenses the architecture into four claims: latency is a design choice, structure should be preserved, cost can be managed through reuse, and production scale is likely to be hybrid.

The slide argues that precomputed KV caches can drive TTFT from seconds toward milliseconds, that caching can materially reduce repeated-input cost, and that CAG can preserve relational integrity that chunked retrieval may lose. It then recommends partitioning knowledge by volatility: stable definitions into Tier 0 caches, dynamic updates into Tier 1 retrievers.

That framing is useful because it stops treating “memory” as one monolithic feature. An LLM system can have several memory classes:

• Model weights for generalized learned capability.
• Stable KV state for reusable, bounded, structurally sensitive knowledge.
• Retrieval indexes for large and volatile external knowledge.
• Runtime state for the current query, tool outputs, and execution feedback.
• Durable application storage for authoritative records that should not depend on model context at all.

The best architecture is the one that puts each fact in the cheapest memory tier that still meets its freshness, integrity, and latency requirements.

Engineering Principles

The architecture ultimately depends on several principles:

1. Partition knowledge by volatility
   Do not cache everything and do not retrieve everything. Stable definitions belong in reusable state; fast-changing evidence belongs in retrieval or authoritative storage.

2. Preserve relationships, not just tokens
   Evaluate whether the system retains foreign keys, constraints, dependency edges, and multi-hop paths. Semantic similarity alone is not enough for structured reasoning.

3. Move reusable work off the critical path
   Prefill, parsing, schema loading, and other deterministic preparation should happen before the request when the inputs are stable.

4. Treat cache state as a versioned production artifact
   Cache contents need ownership, invalidation, warm-up, compatibility checks, and rollback just like indexes or compiled assets.

5. Measure the whole system
   TTFT, cache-hit rate, VRAM residency, rebuild time, retrieval recall, stale-context incidents, and end-to-end task accuracy all matter. Optimizing only one metric can hide failure somewhere else.

Final Synthesis

The most useful conclusion is not that CAG replaces RAG. It is that retrieval should no longer be the default answer for every form of external knowledge.

The comparison infographic in the supplied material summarizes the design space: RAG behaves like a dynamic searcher, while CAG behaves like a preloaded expert. The RAG path provides scale and freshness at the cost of runtime retrieval. The CAG path provides low-latency access to a bounded knowledge base at the cost of memory residency and rebuild complexity. The hybrid path combines them.

The performance snapshot in the infographic reports a 27s standard-RAG prefill path versus an optimized CAG path under 6ms, alongside prompt-caching savings and accuracy improvements. Those are presentation-specific benchmark claims, but the direction of the architecture is clear: remove stable knowledge from repeated runtime work, preserve structure where relationships matter, and reserve retrieval for information that genuinely needs to be discovered at request time.

A production-ready system combines:

• Deterministic foundation: authoritative schemas, libraries, prompts, and stable definitions prepared into a versioned Tier 0 cache.
• Testing layer: multi-hop schema tests, cache-invalidity tests, retrieval-recall tests, fuzzing, and sandboxed execution.
• Agent or AI layer: the LLM reasons over cached context and consumes fresh deltas from Tier 1 retrieval.
• Security and governance: isolated execution, controlled cache rebuilds, versioned knowledge artifacts, and clear separation between model context and authoritative records.
• Observability layer: TTFT, hit/miss rates, cache residency, rebuild latency, retrieval freshness, error categories, and task-level correctness.
• Production outcome: low-latency access to stable structure without sacrificing the scale and freshness of dynamic retrieval.

Closing Thought

The next generation of LLM memory is unlikely to be a single database, a single cache, or a single context window. It will be a hierarchy.

Cache what is stable. Retrieve what is changing. Keep structure intact across both.