Progressive disclosure for catalogs: load-on-demand schemas

[leoafarias]

Is your feature request related to a problem? Please describe.

Right now PromptBuilder inlines the entire A2UI schema into the system prompt on every turn. Every catalog item's full schema goes in, combined into one big oneOf. As catalogs grow, and especially for custom catalogs, that creates two problems.

First, token cost. The whole schema is re-sent on every request, so even a modest 16-item custom catalog already runs well over 10k prompt tokens, and it scales linearly with the catalog. We pay that on every turn.

Second, and more important, the model still uses components that aren't in the catalog. Even with everything inlined it will invent something like Column for a single-item custom catalog, and the SDK throws CatalogItemNotFoundException (#771). That issue was closed by removing the hardcoded standard-catalog examples from the prompt strings, but there's still nothing structural keeping the model to items that actually exist. The reporter put it well: using the SDK with a custom catalog from scratch is "hard / not possible."

Underneath both is the same thing: the model gets every schema at once, with no index in between and no real contract about which components it's actually allowed to use.

Describe the solution you'd like

An opt-in catalog mode built on progressive disclosure, with two tiers:

• A manifest that's always in the prompt: just each item's name and a short description. No full schemas, so it stays cheap at any catalog size.
• An on-demand body: a loadCatalogItems tool the model calls to pull the exact schema and examples for the components it's about to use, before it emits any A2UI. Load before use.

The host registers loadCatalogItems and resolves it against the in-process catalog. If the model asks for a name that doesn't exist, it gets a structured error back and can self-correct on the next turn instead of emitting something unrenderable. The current full-schema behavior stays the default; this is opt-in.

Why I think this helps:

• Fewer tokens: the per-turn prompt only carries the manifest, so input drops sharply. Full schemas are paid once, on demand, and only for what's actually used.
• More purposeful context, not just less of it. With "load before use" the prompt only ever holds the components actually in play, instead of every schema at once. That's cheaper, but the bigger win is signal: a focused, high-signal context is easier for the model to reason over than a large one padded with schemas it will never use. There's good research behind this (see Additional context). On top of that it makes the Gemini trying to render items that are not in the catalog #771 failure structural to avoid, since the model has to name a real component and receive its real schema before it can use it, rather than being kept in line by prompt wording alone.
• It scales to large and custom catalogs, which is exactly where the current approach hurts most.

This does depend on the catalog id being available in the prompt, since createSurface needs it.

Describe alternatives you've considered

• Keep inlining the full schema and lean on prompt wording to keep the model in bounds (today's approach, and Gemini trying to render items that are not in the catalog #771's fix). Doesn't scale on tokens, and gives no guarantee against invented components.
• Trim the inlined schema heuristically, e.g. only the "likely" items. Fragile, guesses intent, and still paid on every turn.
• Retrieval / RAG over catalog items. Heavier infrastructure than a deterministic tool call for an in-process catalog.
• A static, name-only allow-list in the prompt. Tells the model the names but not how to use them, so the schema still has to live somewhere.

Additional context

• This mirrors Anthropic's Agent Skills progressive disclosure: name and description are always loaded, and the full body is loaded on demand.
• Why "purposeful loading" helps beyond saving tokens: there's evidence that excess or irrelevant context measurably degrades LLM reasoning (Shi et al., 2023), and that models use information worse as it gets buried in a longer context — the "lost in the middle" effect (Liu et al., 2023). The practitioner takeaway, from both Anthropic and Google, is to keep prompts high-signal and pull detail in just-in-time via tools rather than front-loading everything (Anthropic, Gemini function calling).
• Initial validation (small, directional): on the simple_chat custom catalog with gemini-flash-latest, incremental used ~60% fewer tokens at the same expectation pass-rate. There is a latency spike from the extra loadCatalogItems round-trip, so it trades latency for tokens. A fuller eval across more prompts and models would firm this up.
• Related issues: Gemini trying to render items that are not in the catalog #771 (renders items not in the catalog), Add descriptions to schemas for all catalog items to improve generation quality #554 (schema descriptions, which the manifest would rely on), PromptBuilder doesn't surface the Catalog's id #900 (surfacing the catalog id, which incremental needs).

[gspencergoog]

If I can summarize your proposal, it seems you are proposing to:

• Reduce the catalog to a manifest of the available catalog items (including client side functions, I assume) with descriptions and use cases (like a skill frontmatter).
• Provide a tool inference that will be given a list of catalog schemas to retrieve, and return a full schema for the model to use when constructing A2UI output.

This is an approach we have also considered, and it is a reasonable solution. The down side is that it requires an extra inference in order to make the tool call to load the catalog items. This isn't necessarily bad, but it is a latency cost. It may be better than the latency cost of a large context, but that needs to be benchmarked for a variety of use cases.

The effectiveness also depends a lot on the descriptions given in the manifest, and what information each entry contains.

[leoafarias]

Thanks, that summary matches the proposal: keep a compact catalog manifest in the prompt, then use loadCatalogItems to fetch full schemas/examples only for components the model is about to render.

I ran a small local smoke test to check the tradeoff. Directionally, the generated UI was equivalent, but the resource profile changed.

With gemini-flash-latest, thinking off, with the chat demo catalog.

• Incremental used ~3.9k mean total tokens vs ~11.9k for full-schema, about a 67% reduction.
• Latency was slightly higher for incremental: ~3.9s vs ~3.3s.
• Incremental loaded every rendered component before use.

One caveat: thinking config matters. In an earlier run, Gemini dynamic thinking dominated latency for both modes. Since incremental adds extra model/tool round trips, hidden thinking can make the schema-loading cost look larger or harder to attribute. Any benchmark should hold that setting constant.

So I agree the latency concern is real. The tradeoff seems to be lower token cost and a checkable load-before-use contract, in exchange for extra round-trip latency.

Given that, I can put together a narrow PR for review:

• full-schema remains the default
• incremental is explicit opt-in, e.g. CatalogPromptMode.incremental
• incremental emits a manifest instead of the full schema
• hosts register loadCatalogItems
• no broader catalog refactor, example wiring, or benchmark tooling

That would let us review the API shape and prompt/tool contract separately from broader benchmarking. What do you think?

[leoafarias]

I created a PR to explore a possible implementation, as well as any related evals.

I created some harness for validation and tests, but did not add it to the PR because there is much more harness eval code than implementation itself.

The token win held: ~60% fewer total tokens for incremental, with equal or better output quality, and it loaded every component before using it. This ~60% is using the basic component built into the chat demo. In our component library, we see closer to 80-90%, depending on whether we are loading only primitives or full UI patterns.

On the latency side, latency becomes a problem with the added "thinking" the model must do per turn. However, without thinking, latency is around ~300ms, so the cost is small, within roughly half a second of full-schema. Most of what looked like a larger penalty earlier was hidden model thinking, not the extra turn; with thinking off, both modes sit at a few seconds.

On the manifest descriptions, agreed, that's the key lever, and it's the least tested one here, so I'd treat stress-testing sparse or weak descriptions as a follow-up rather than something these numbers cover.

In the implementation, full-schema stays the default, incremental is opt-in, and I'd lean toward selecting the mode deterministically (small catalogs vs. large/custom) rather than trying to control round-trips through the prompt.

Now a few benefits of incremental that we also saw:

• No overload errors from the latest Gemini models; for some reason, this completely removed all errors we were getting from model overload with the latest Gemini.
• We can be more detailed about examples and instructions on how to use a component without worrying about the initial full component catalog load.

[gspencergoog]

The advantages of this method as I see them:

• Larger catalogs become more manageable (or possible, depending on your definition of "large")
• Reduced context size improves results and decreases model distraction.
• Lower token usage reduces inference cost (in $$).

The disadvantages are:

• Slightly higher latency
• Slightly increased complexity, since you have to implement and configure tools for component selection
• The possibility for incorrect component selection.
  • For example, if the model needs a Card with a submit button on it, and there are three cards in the catalog, but none of them mention in their description that they have a submit button, it picks randomly. But if it had the whole schema up front, it would be able to tell which one had a submit button.
  • Basically, if it needs to know details of the components that aren't present in the descriptions, it could fail to select the right component.
• Possible increased $$ cost because of extra tool call/inference/tokens in the tool call. This may or may not offset the cost difference from lower token usage in the input, it's not clear.

These are not horrible downsides, and the ability to have large catalogs (perhaps VERY large) is a nice advantage. I suspect that it is quite use case dependent whether or not this strategy is the best strategy, though.

If latency isn't a main concern, then you could imagine similar strategies that might scale even more. Consider that instead of incorporating the component descriptions in the prompt, you could provide a tool that searches a vector database of components and their descriptions. This would add the same extra inference, but could scale to millions of components and would require only the tool declaration token overhead for the vector database lookup. It might require some prompt engineering to get the model to send the right kind of query to the vector database, but it could be quite effective.

[leoafarias]

Thanks for looking into it. I do think there is one benefit we are seeing that I am not sure how to quantify. But the quality of component generation goes up, and hallucination goes down. I am still trying to get more data here to get some more measurable impact, so we can have some test cases that we can run and compare.