Generative UI

Some answers are awkward to give in text. For example, the agent may need you to enter an amount or choose an item from a list, or it may have data that is easier to read as a card. In these cases the agent can reply with a piece of UI.

It works like this. The agent writes a description of the UI in JSON, which we call a spec. The app takes the matching native components from a fixed catalog and shows the UI. This system is built on json-render.

Component props are semantic, such as tone and size, and never raw colour values. As a result, anything the agent builds looks like the rest of the app. The agent also uses this catalog when it builds dashboard widgets, so it only has to learn one way of describing UI.

The specCopy Anchor Link

// specSchema in @pku-harness/sdk (genui.ts)
type UISpec = {
  root: string;                                  // id of the root element
  elements: Record<string, {
    type: string;                                // a catalog component
    props: Record<string, unknown>;              // every listed prop present; null when unused
    children?: string[];                         // element ids
    on?: Record<string, unknown>;                // events, e.g. { press: { action, params } }
    visible?: unknown;
  }>;
  state?: Record<string, unknown>;               // initial state, e.g. { form: { amount: "50" } }
};

A spec can hold some state (state). Component props can be bound to that state in three ways.

CatalogCopy Anchor Link

The catalog has 17 components. Components that "have children" contain other components through children. "Bind value" means the input's value should be bound to state with $bindState.

When a component is tapped, it runs the action defined in on.press. There are two actions.

A tool run by callTool goes through the same permission check as a call from the agent.

This page may fall behind the code, but the catalog at runtime is always current. The description of the ui.render tool includes the output of services.genui.describeCatalog(), so the agent always sees the current catalog. You can validate a spec with services.genui.checkSpec(spec).

Example: a top-up formCopy Anchor Link

This spec shows a "充值校园卡" (top up campus card) card with an amount choice and a confirm button. When you tap the button, the chosen amount goes back to the agent as your next message.

{
  "root": "card",
  "elements": {
    "card": { "type": "Card", "props": { "title": "充值校园卡", "subtitle": "选择金额" }, "children": ["amount", "ok"] },
    "amount": {
      "type": "Choice",
      "props": {
        "label": null,
        "options": [{ "label": "¥50", "value": "50", "hint": null }, { "label": "¥100", "value": "100", "hint": "常用" }],
        "value": { "$bindState": "/form/amount" }
      }
    },
    "ok": { "type": "Button", "props": { "label": "确认", "variant": "primary" }, "on": { "press": { "action": "submit", "params": { "label": "充值" } } } }
  },
  "state": { "form": { "amount": "100" } }
}

For plugin authorsCopy Anchor Link

Remote plugins can't show UI themselves. They return data, and the agent builds the UI.

Screens in bundled plugins can show a spec with host.components.GenUI: <GenUI spec={spec} onSubmit={(form, label) => …} />.

Generative UI suits one-off moments in a conversation. If users will open something often, build a native screen or widget for it instead (contributes.screens / widgets).