Skip to content

This page is taken from the examples/widgets/ directory and refers to the code there.

Dynamic Widgets Demo 📊

This demo shows how async tools can be used to enrich model output with UI elements.

License

Each UI element (e.g., a plot, map, 3D viewer) is registered as a tool. In this case, these are ImageGenerator (backed by Imagen 3) and PlotGenerator (backed by Gemini instructed to output SVG).

  • The model outputs the element by invoking the corresponding tool. This way we piggy-back on all security features baked into function calling such as resilience to prompt injection and escaping attacks.
  • Tools are async and thus do not block the model from proceeding to output the rest of the reply. In fact, they immediately respond with a will_continue=False message notifying the model that it won't hear back from these tools. A future optimization would be to make the model continue generating without interruption for the tool call.
  • The tool streams the response directly to the client by sending ProcessorParts with substream_name=status. This is a "reserved" substream which is routed straight to the client and doesn't get sent to the model.
  • The client can use part.function_response.id to associate responses to the original FunctionCall part. This allows us to avoid head-of-line blocking: the model can continue producing an output without waiting for the widgets to be generated.

Note that this way we create a tiered approach to widget rendering:

  • Model decides which widget needs to be rendered and what content it should have.
  • Tool is responsible for rendering the widget into HTML or another "low level" representation.
  • UI is responsible for displaying this HTML in the right place.

This setup allows streaming content independently of widgets and rendering widgets in parallel. We can even stream widgets themselves.

🚀 How to run it

This example comes with a web UI based on AI Studio Applets.

  1. Install the dependencies with pip install genai-processors[live].

  2. Define a GOOGLE_API_KEY environment variable with your API key.

  3. Launch the widgets agent: 

    python3 -m genai_processors.examples.widgets.widgets_ais

  4. Access the AI studio applet . The applet source is located in ais_app/.

  5. Try the prompt: > Describe what conic sections are illustrating each of them with a plot.

    Note how the browser renders plots while they are being generated.

⚠️ Limitations

  • The prompts were not optimized for quality: the agent's output may contain errors. The purpose of this demo is to demonstrate how to render widgets, not how to generate good ones.
  • Illustrations and plots are not conditioned to adhere to a specific style.