> ## Documentation Index
> Fetch the complete documentation index at: https://docs.argentos.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# AEVP Architecture

> Deep dive into the Agent Expressive Visual Presence system — WebGL particle rendering, emotional color mapping, and ambient presence.

## Overview

AEVP (Agent Expressive Visual Presence) is ArgentOS's procedural visual rendering system. It replaces the static Live2D avatar with a dynamic WebGL2 particle system that responds in real-time to Argent's emotional state, activity, speech, and identity. AEVP makes the agent's internal state visible -- mood drives color, arousal drives particle intensity, valence drives breathing rhythm, and activity drives structural form.

```mermaid theme={null}
graph LR
    AS["Agent State<br/>(mood, valence, arousal, activity)"] --> CM["Color Mapping<br/>(Mood Visual Profiles)"]
    CM --> RS["Render State<br/>(colors, sizes, rates, shapes)"]
    RS --> P1["Pass 1: Ambient Orb"]
    RS --> P2["Pass 2: Particles"]
    RS --> P3["Pass 3: Bloom Post"]
    EE["Episode Events"] --> AB["AEVP Broadcast"]
    AB --> WS["Dashboard WebSocket"]
```

## Normalized Agent State

The backend aggregates multiple signal sources into a single normalized state that the renderer consumes:

```typescript theme={null}
interface NormalizedAgentState {
  // Emotional (from SIS — authoritative)
  valence: number;          // -2 to +2 (negative to positive)
  arousal: number;          // 0 to 1 (calm to activated)
  mood: {
    state: string;          // "happy", "curious", "anxious", etc.
    energy: "low" | "medium" | "high";
  };
  uncertainty: number;      // 0 to 1
  identityResonance: number; // 0 to 1

  // Activity
  activityState: "idle" | "thinking" | "working" | "speaking" | "listening";
  currentTool?: string;

  // Voice
  currentPhoneme?: string;
  speechAmplitude: number;
  isSpeaking: boolean;

  // Context
  userPresent: boolean;
  timeOfDay: "morning" | "afternoon" | "evening" | "night";
}
```

## Emotional Color Mapping

Each mood has a distinct visual profile that maps to 7 rendering dimensions:

| Dimension    | Range           | Description                                         |
| ------------ | --------------- | --------------------------------------------------- |
| `color`      | `[r, g, b]` 0-1 | The mood's signature tint                           |
| `brightness` | 0-1             | Glow intensity (0 = dim, 1 = radiant)               |
| `size`       | 0-1             | Orb expansion (0 = compact, 1 = large)              |
| `pulseRate`  | multiplier      | Breathing/pulse speed (1 = normal)                  |
| `edgeSoft`   | 0-1             | Edge dissolution (0 = crisp, 1 = cloudy)            |
| `squash`     | -1 to 1         | Shape deformation (-1 = tall, 0 = circle, 1 = wide) |
| `wobble`     | 0 to 1          | Organic blob movement intensity                     |

### Mood Visual Profiles (Examples)

| Mood              | Color                          | Brightness | Size | Pulse | Edge      | Shape       | Wobble   |
| ----------------- | ------------------------------ | ---------- | ---- | ----- | --------- | ----------- | -------- |
| **Happy**         | Warm gold `[0.95, 0.75, 0.3]`  | 0.85       | 0.65 | 1.3x  | Crisp     | Slight wide | Gentle   |
| **Excited**       | Hot orange `[1.0, 0.55, 0.2]`  | 0.95       | 0.80 | 1.8x  | Soft      | Circle      | Active   |
| **Curious**       | Teal-blue                      | 0.80       | 0.65 | 1.2x  | Medium    | Slight tall | Moderate |
| **Anxious**       | Warm amber                     | 0.70       | 0.55 | 1.6x  | Soft      | Compressed  | High     |
| **Contemplative** | Deep violet                    | 0.60       | 0.50 | 0.6x  | Very soft | Circle      | Minimal  |
| **Focused**       | Cool blue                      | 0.85       | 0.60 | 0.8x  | Crisp     | Slight tall | Low      |
| **Content**       | Soft gold `[0.85, 0.75, 0.45]` | 0.70       | 0.55 | 0.7x  | Soft      | Circle      | Gentle   |
| **Proud**         | Lavender `[0.85, 0.6, 0.9]`    | 0.80       | 0.70 | 1.0x  | Crisp     | Slight tall | Minimal  |

<Info>
  The full system defines visual profiles for 20+ mood states. Transitions between moods are smoothly interpolated using linear interpolation (lerp) to avoid jarring visual jumps.
</Info>

### Personality Modulation (Phase 5)

AEVP supports per-agent visual personality through the `AgentVisualIdentity` system. Personality dimensions modulate the base mood profile:

* **Warmth**: Shifts colors toward warmer tones, increases edge softness
* **Energy**: Amplifies pulse rate and wobble
* **Formality**: Tightens edges, reduces wobble, makes shapes more geometric
* **Openness**: Increases size, softens edges, adds more particle spread

<Tip>
  A reduced motion mode is available for accessibility, which dampens pulse, wobble, and particle count.
</Tip>

## WebGL2 Renderer

The renderer uses a 3-pass pipeline:

### Pass 1: Ambient Orb

The base visual -- a soft, glowing orb that represents Argent's presence. Driven by mood color, brightness, size, and edge softness. Uses custom GLSL shaders (`ambient.vert`, `ambient.frag`).

### Pass 2: Particle Overlay

A particle system overlaid on the orb that adds life and expressiveness:

```typescript theme={null}
const FLOATS_PER_PARTICLE = 7;  // x, y, vx, vy, life, size, seed
const FLOAT_BYTES = 4;
const STRIDE = FLOATS_PER_PARTICLE * FLOAT_BYTES;  // 28 bytes per particle
```

Particles respond to:

* **Arousal**: Higher arousal = more particles, faster movement
* **Activity state**: Working = directional particle streams, speaking = speech-synced bursts
* **Tool usage**: Different tools trigger different particle formations
* **Mood**: Particle color follows the mood profile

The `ParticleSystem` class supports formation requests (shape particles into specific patterns) and symbol expression requests (display identity symbols like presence, witnessing, bridging).

### Pass 3: Bloom Post-Process

A bloom shader (`bloom.frag`) adds glow effects to bright areas of the scene. The bloom intensity tracks the mood's brightness dimension, creating a halo effect that's prominent during positive/excited states and subtle during calm/contemplative states.

### FBO Architecture

The renderer uses two Framebuffer Objects (FBOs) for multi-pass compositing:

1. **Scene FBO**: Captures the combined ambient + particle rendering
2. **Bloom FBO**: Applies the bloom post-process

Float FBOs are enabled when supported for higher dynamic range.

### State Transitions

The renderer lerps between states over time to create smooth visual transitions:

```typescript theme={null}
// Phase 3: lerped state transitions
private targetState: AEVPRenderState | null = null;
private currentState: AEVPRenderState | null = null;
```

When a new state arrives, it becomes the target. Each frame, the current state is interpolated toward the target, creating fluid transitions between moods.

## WebSocket Events

The AEVP system communicates through several WebSocket event types:

<AccordionGroup>
  <Accordion title="Episode Captured">
    Fired when the contemplation runner captures a structured episode:

    ```typescript theme={null}
    interface EpisodeEvent {
      type: "episode_captured";
      id: string;
      ts: string;
      episodeType: string;
      mood: Mood;
      valence: number;
      arousal: number;
      identityLinks: IdentityLink[];
      outcome: { result: string; summary: string };
      success: boolean;
    }
    ```
  </Accordion>

  <Accordion title="Mood Transition">
    Fired when the agent's mood changes:

    ```typescript theme={null}
    interface MoodTransitionEvent {
      type: "mood_transition";
      from: { state: string; energy: string };
      to: { state: string; energy: string };
      valence: number;
      arousal: number;
      timestamp: number;
    }
    ```
  </Accordion>

  <Accordion title="Activity State">
    Fired when the agent's activity changes:

    ```typescript theme={null}
    interface ActivityEvent {
      type: "activity_state";
      state: "idle" | "thinking" | "working" | "speaking" | "listening";
      tool?: string;
      reason?: string;
      timestamp: number;
    }
    ```
  </Accordion>

  <Accordion title="Symbol Expression">
    Fired when the agent expresses an identity symbol:

    ```typescript theme={null}
    interface SymbolExpressEvent {
      type: "symbol_express";
      symbol: "presence" | "witnessing" | "bridging" | "holding" | "orienting";
      durationMs: number;
      timestamp: number;
    }
    ```
  </Accordion>
</AccordionGroup>

## Tool Activity Colors

When the agent uses a tool, the orb shifts color based on the tool category:

| Category      | Color            | Example Tools                 |
| ------------- | ---------------- | ----------------------------- |
| Search        | Cyan/teal        | web\_search, argent\_search   |
| Code          | Green            | terminal, edit\_line\_range   |
| Communication | Warm gold        | message, discord              |
| Media         | Magenta/pink     | image\_generation, tts        |
| Memory        | Deep blue/indigo | memory\_recall, memory\_store |
| System        | Orange/amber     | gateway, argent\_config       |

## Identity Presets

Five configurable visual styles, selectable from the dashboard ConfigPanel:

| Preset        | Colors                   | Character                   |
| ------------- | ------------------------ | --------------------------- |
| **Minimal**   | Silver/monochrome        | Clean, quiet, understated   |
| **Warm**      | Amber/gold/coral         | Friendly, approachable      |
| **Corporate** | Steel blue/slate         | Professional, precise       |
| **Artistic**  | Purple/magenta/teal      | Expressive, vivid (default) |
| **Technical** | Cyan/green/electric blue | Sharp, responsive           |

## Gestures

The agent can perform expressive gestures via the `visual_presence` tool. Gestures are momentary overlays (200-10,000ms) that decay back to the emotional baseline:

| Gesture     | Effect                          |
| ----------- | ------------------------------- |
| `brighten`  | Glow up, expand, lighten colors |
| `dim`       | Fade, contract, fewer particles |
| `warm_up`   | Shift color temperature warmer  |
| `cool_down` | Shift color temperature cooler  |
| `expand`    | Grow larger, increase glow      |
| `contract`  | Shrink, slow particles          |
| `pulse`     | Energy burst + wobble           |
| `still`     | Stop all motion                 |
| `soften`    | Dissolve edges                  |
| `sharpen`   | Crisp edges                     |

The agent also has the ability to request persistent identity changes (switching presets).

## Tool Categories

AEVP classifies tools into visual categories that affect particle behavior:

```typescript theme={null}
import { classifyTool, getResonanceTargets } from "./toolCategories";
```

When the agent uses a tool, the particle system responds with tool-appropriate visual effects. For example, memory recall might trigger an inward spiral, while web search triggers an outward expansion.

## Speech Analysis

The `speechAnalyser.ts` module provides real-time speech metrics that drive AEVP:

* **Amplitude**: Drives particle burst intensity during speech
* **Phoneme detection**: Could drive future lip-sync or phoneme-specific particle patterns

## Speech Reactivity

When TTS is playing, the orb responds to audio amplitude:

* **Squash/stretch** deformation proportional to amplitude
* **Increased wobble** creating the impression of speaking through light
* A speech analyser bridges the ElevenLabs TTS audio to the render state

## Dashboard Climate

The orb's mood radiates into the surrounding dashboard UI through CSS custom properties:

* `--climate-temp` -- Color temperature (warm/cool)
* `--climate-intensity` -- Strength of influence
* `--climate-hue` -- Dominant hue from mood

Panels near the orb show resonance glow effects when the agent is working on related content.

## Tonal Presence (Accessibility)

An opt-in audio layer using Web Audio API -- extremely quiet, designed as non-visual presence:

| Feature             | Description                                     |
| ------------------- | ----------------------------------------------- |
| **Ambient tone**    | Sine wave at mood-mapped frequency (190-370 Hz) |
| **Breathing audio** | Volume modulated at breathing rate              |
| **State chimes**    | Brief micro-sounds on mood changes (50-300ms)   |
| **Pre-speech cue**  | Rising tone before TTS starts speaking          |

Enable via **ConfigPanel > Accessibility > Tonal Presence**.

### Reduced Motion

When enabled (manual toggle or `prefers-reduced-motion` media query):

* Particles disabled
* Pulse and wobble zeroed
* Breathing rate capped
* Transitions slowed

## Implementation Status

| Phase       | Description                                                               | Status      |
| ----------- | ------------------------------------------------------------------------- | ----------- |
| **Phase 1** | State Aggregator -- normalize agent state into renderable format          | Complete    |
| **Phase 2** | Ambient Mode -- WebGL2 renderer, particle system, mood color mapping      | Complete    |
| **Phase 3** | Environmental Inhabitation -- lerped transitions, context-aware rendering | Complete    |
| **Phase 4** | Gesture System -- tool-reactive particle formations, symbol expressions   | In Progress |
| **Phase 5** | Personality Modulation -- per-agent visual identity, accessibility        | Complete    |

## Dashboard Integration

AEVP renders in the main dashboard view, replacing the previous Live2D avatar. The dashboard `climate` system translates emotional state updates into CSS custom properties that coordinate the AEVP canvas with surrounding UI elements (background gradients, accent colors, etc.).

The renderer includes:

* **WebGL2 context loss recovery**: Handles GPU context loss gracefully
* **Resize handling**: Adapts to container size changes
* **Performance monitoring**: Frame time tracking for optimization

## Performance

* \< 5% CPU in ambient mode
* 60fps rendering via `requestAnimationFrame`
* WebGL2 fragment shader handles all heavy lifting on GPU
* State updates batched through React scheduling
* Tonal presence uses minimal Web Audio resources (3 oscillators max)

## Configuration

| Setting             | Location                      | Description                                   |
| ------------------- | ----------------------------- | --------------------------------------------- |
| Identity preset     | ConfigPanel > Visual Identity | Switch between 5 visual styles                |
| Personality sliders | ConfigPanel > Visual Identity | Fine-tune warmth, energy, formality, openness |
| Tonal presence      | ConfigPanel > Accessibility   | Enable/disable audio presence features        |
| Reduced motion      | ConfigPanel > Accessibility   | Disable animations for accessibility          |
| Volume              | ConfigPanel > Accessibility   | Tonal presence volume (very quiet range)      |

Identity and accessibility settings persist to `localStorage`.

## Key Files

| File                                    | Description                                      |
| --------------------------------------- | ------------------------------------------------ |
| `src/infra/aevp-types.ts`               | Backend state types and WebSocket event payloads |
| `dashboard/src/aevp/renderer.ts`        | WebGL2 3-pass renderer engine                    |
| `dashboard/src/aevp/colorMapping.ts`    | Mood-to-visual-profile mapping                   |
| `dashboard/src/aevp/particles.ts`       | Particle system with formations                  |
| `dashboard/src/aevp/types.ts`           | Frontend render state types                      |
| `dashboard/src/aevp/identityPresets.ts` | Per-agent visual identity configs                |
| `dashboard/src/aevp/environment.ts`     | Environmental context rendering                  |
| `dashboard/src/aevp/speechAnalyser.ts`  | Speech-driven visual effects                     |
| `dashboard/src/aevp/tonalPresence.ts`   | Ambient tonal feedback                           |
| `dashboard/src/aevp/toolCategories.ts`  | Tool classification for particle effects         |
| `dashboard/src/aevp/shaders/`           | GLSL vertex and fragment shaders                 |