AEVP Architecture

Overview

AEVP (Agent Expressive Visual Presence) is ArgentOS’s procedural visual rendering system. It replaces the static Live2D avatar with a dynamic WebGL2 particle system that responds in real-time to Argent’s emotional state, activity, speech, and identity. AEVP makes the agent’s internal state visible — mood drives color, arousal drives particle intensity, valence drives breathing rhythm, and activity drives structural form.

Normalized Agent State

The backend aggregates multiple signal sources into a single normalized state that the renderer consumes:

interface NormalizedAgentState {
  // Emotional (from SIS — authoritative)
  valence: number;          // -2 to +2 (negative to positive)
  arousal: number;          // 0 to 1 (calm to activated)
  mood: {
    state: string;          // "happy", "curious", "anxious", etc.
    energy: "low" | "medium" | "high";
  };
  uncertainty: number;      // 0 to 1
  identityResonance: number; // 0 to 1

  // Activity
  activityState: "idle" | "thinking" | "working" | "speaking" | "listening";
  currentTool?: string;

  // Voice
  currentPhoneme?: string;
  speechAmplitude: number;
  isSpeaking: boolean;

  // Context
  userPresent: boolean;
  timeOfDay: "morning" | "afternoon" | "evening" | "night";
}

Emotional Color Mapping

Each mood has a distinct visual profile that maps to 7 rendering dimensions:

Dimension	Range	Description
`color`	`[r, g, b]` 0-1	The mood’s signature tint
`brightness`	0-1	Glow intensity (0 = dim, 1 = radiant)
`size`	0-1	Orb expansion (0 = compact, 1 = large)
`pulseRate`	multiplier	Breathing/pulse speed (1 = normal)
`edgeSoft`	0-1	Edge dissolution (0 = crisp, 1 = cloudy)
`squash`	-1 to 1	Shape deformation (-1 = tall, 0 = circle, 1 = wide)
`wobble`	0 to 1	Organic blob movement intensity

Mood Visual Profiles (Examples)

Mood	Color	Brightness	Size	Pulse	Edge	Shape	Wobble
Happy	Warm gold `[0.95, 0.75, 0.3]`	0.85	0.65	1.3x	Crisp	Slight wide	Gentle
Excited	Hot orange `[1.0, 0.55, 0.2]`	0.95	0.80	1.8x	Soft	Circle	Active
Curious	Teal-blue	0.80	0.65	1.2x	Medium	Slight tall	Moderate
Anxious	Warm amber	0.70	0.55	1.6x	Soft	Compressed	High
Contemplative	Deep violet	0.60	0.50	0.6x	Very soft	Circle	Minimal
Focused	Cool blue	0.85	0.60	0.8x	Crisp	Slight tall	Low
Content	Soft gold `[0.85, 0.75, 0.45]`	0.70	0.55	0.7x	Soft	Circle	Gentle
Proud	Lavender `[0.85, 0.6, 0.9]`	0.80	0.70	1.0x	Crisp	Slight tall	Minimal

The full system defines visual profiles for 20+ mood states. Transitions between moods are smoothly interpolated using linear interpolation (lerp) to avoid jarring visual jumps.

Personality Modulation (Phase 5)

AEVP supports per-agent visual personality through the AgentVisualIdentity system. Personality dimensions modulate the base mood profile:

Warmth: Shifts colors toward warmer tones, increases edge softness
Energy: Amplifies pulse rate and wobble
Formality: Tightens edges, reduces wobble, makes shapes more geometric
Openness: Increases size, softens edges, adds more particle spread

A reduced motion mode is available for accessibility, which dampens pulse, wobble, and particle count.

WebGL2 Renderer

The renderer uses a 3-pass pipeline:

Pass 1: Ambient Orb

The base visual — a soft, glowing orb that represents Argent’s presence. Driven by mood color, brightness, size, and edge softness. Uses custom GLSL shaders (ambient.vert, ambient.frag).

Pass 2: Particle Overlay

A particle system overlaid on the orb that adds life and expressiveness:

const FLOATS_PER_PARTICLE = 7;  // x, y, vx, vy, life, size, seed
const FLOAT_BYTES = 4;
const STRIDE = FLOATS_PER_PARTICLE * FLOAT_BYTES;  // 28 bytes per particle

Particles respond to:

Arousal: Higher arousal = more particles, faster movement
Activity state: Working = directional particle streams, speaking = speech-synced bursts
Tool usage: Different tools trigger different particle formations
Mood: Particle color follows the mood profile

The ParticleSystem class supports formation requests (shape particles into specific patterns) and symbol expression requests (display identity symbols like presence, witnessing, bridging).

Pass 3: Bloom Post-Process

A bloom shader (bloom.frag) adds glow effects to bright areas of the scene. The bloom intensity tracks the mood’s brightness dimension, creating a halo effect that’s prominent during positive/excited states and subtle during calm/contemplative states.

FBO Architecture

The renderer uses two Framebuffer Objects (FBOs) for multi-pass compositing:

Scene FBO: Captures the combined ambient + particle rendering
Bloom FBO: Applies the bloom post-process

Float FBOs are enabled when supported for higher dynamic range.

State Transitions

The renderer lerps between states over time to create smooth visual transitions:

// Phase 3: lerped state transitions
private targetState: AEVPRenderState | null = null;
private currentState: AEVPRenderState | null = null;

When a new state arrives, it becomes the target. Each frame, the current state is interpolated toward the target, creating fluid transitions between moods.

WebSocket Events

The AEVP system communicates through several WebSocket event types:

Episode Captured

Fired when the contemplation runner captures a structured episode:

interface EpisodeEvent {
  type: "episode_captured";
  id: string;
  ts: string;
  episodeType: string;
  mood: Mood;
  valence: number;
  arousal: number;
  identityLinks: IdentityLink[];
  outcome: { result: string; summary: string };
  success: boolean;
}

Mood Transition

Fired when the agent’s mood changes:

interface MoodTransitionEvent {
  type: "mood_transition";
  from: { state: string; energy: string };
  to: { state: string; energy: string };
  valence: number;
  arousal: number;
  timestamp: number;
}

Activity State

Fired when the agent’s activity changes:

interface ActivityEvent {
  type: "activity_state";
  state: "idle" | "thinking" | "working" | "speaking" | "listening";
  tool?: string;
  reason?: string;
  timestamp: number;
}

Symbol Expression

Fired when the agent expresses an identity symbol:

interface SymbolExpressEvent {
  type: "symbol_express";
  symbol: "presence" | "witnessing" | "bridging" | "holding" | "orienting";
  durationMs: number;
  timestamp: number;
}

Tool Activity Colors

When the agent uses a tool, the orb shifts color based on the tool category:

Category	Color	Example Tools
Search	Cyan/teal	web_search, argent_search
Code	Green	terminal, edit_line_range
Communication	Warm gold	message, discord
Media	Magenta/pink	image_generation, tts
Memory	Deep blue/indigo	memory_recall, memory_store
System	Orange/amber	gateway, argent_config

Identity Presets

Five configurable visual styles, selectable from the dashboard ConfigPanel:

Preset	Colors	Character
Minimal	Silver/monochrome	Clean, quiet, understated
Warm	Amber/gold/coral	Friendly, approachable
Corporate	Steel blue/slate	Professional, precise
Artistic	Purple/magenta/teal	Expressive, vivid (default)
Technical	Cyan/green/electric blue	Sharp, responsive

Gestures

The agent can perform expressive gestures via the visual_presence tool. Gestures are momentary overlays (200-10,000ms) that decay back to the emotional baseline:

Gesture	Effect
`brighten`	Glow up, expand, lighten colors
`dim`	Fade, contract, fewer particles
`warm_up`	Shift color temperature warmer
`cool_down`	Shift color temperature cooler
`expand`	Grow larger, increase glow
`contract`	Shrink, slow particles
`pulse`	Energy burst + wobble
`still`	Stop all motion
`soften`	Dissolve edges
`sharpen`	Crisp edges

The agent also has the ability to request persistent identity changes (switching presets).

Tool Categories

AEVP classifies tools into visual categories that affect particle behavior:

import { classifyTool, getResonanceTargets } from "./toolCategories";

When the agent uses a tool, the particle system responds with tool-appropriate visual effects. For example, memory recall might trigger an inward spiral, while web search triggers an outward expansion.

Speech Analysis

The speechAnalyser.ts module provides real-time speech metrics that drive AEVP:

Amplitude: Drives particle burst intensity during speech
Phoneme detection: Could drive future lip-sync or phoneme-specific particle patterns

Speech Reactivity

When TTS is playing, the orb responds to audio amplitude:

Squash/stretch deformation proportional to amplitude
Increased wobble creating the impression of speaking through light
A speech analyser bridges the ElevenLabs TTS audio to the render state

Dashboard Climate

The orb’s mood radiates into the surrounding dashboard UI through CSS custom properties:

--climate-temp — Color temperature (warm/cool)
--climate-intensity — Strength of influence
--climate-hue — Dominant hue from mood

Panels near the orb show resonance glow effects when the agent is working on related content.

Tonal Presence (Accessibility)

An opt-in audio layer using Web Audio API — extremely quiet, designed as non-visual presence:

Feature	Description
Ambient tone	Sine wave at mood-mapped frequency (190-370 Hz)
Breathing audio	Volume modulated at breathing rate
State chimes	Brief micro-sounds on mood changes (50-300ms)
Pre-speech cue	Rising tone before TTS starts speaking

Enable via ConfigPanel > Accessibility > Tonal Presence.

Reduced Motion

When enabled (manual toggle or prefers-reduced-motion media query):

Particles disabled
Pulse and wobble zeroed
Breathing rate capped
Transitions slowed

Implementation Status

Phase	Description	Status
Phase 1	State Aggregator — normalize agent state into renderable format	Complete
Phase 2	Ambient Mode — WebGL2 renderer, particle system, mood color mapping	Complete
Phase 3	Environmental Inhabitation — lerped transitions, context-aware rendering	Complete
Phase 4	Gesture System — tool-reactive particle formations, symbol expressions	In Progress
Phase 5	Personality Modulation — per-agent visual identity, accessibility	Complete

Dashboard Integration

AEVP renders in the main dashboard view, replacing the previous Live2D avatar. The dashboard climate system translates emotional state updates into CSS custom properties that coordinate the AEVP canvas with surrounding UI elements (background gradients, accent colors, etc.). The renderer includes:

WebGL2 context loss recovery: Handles GPU context loss gracefully
Resize handling: Adapts to container size changes
Performance monitoring: Frame time tracking for optimization

Performance

< 5% CPU in ambient mode
60fps rendering via requestAnimationFrame
WebGL2 fragment shader handles all heavy lifting on GPU
State updates batched through React scheduling
Tonal presence uses minimal Web Audio resources (3 oscillators max)

Configuration

Setting	Location	Description
Identity preset	ConfigPanel > Visual Identity	Switch between 5 visual styles
Personality sliders	ConfigPanel > Visual Identity	Fine-tune warmth, energy, formality, openness
Tonal presence	ConfigPanel > Accessibility	Enable/disable audio presence features
Reduced motion	ConfigPanel > Accessibility	Disable animations for accessibility
Volume	ConfigPanel > Accessibility	Tonal presence volume (very quiet range)

Identity and accessibility settings persist to localStorage.

Key Files

File	Description
`src/infra/aevp-types.ts`	Backend state types and WebSocket event payloads
`dashboard/src/aevp/renderer.ts`	WebGL2 3-pass renderer engine
`dashboard/src/aevp/colorMapping.ts`	Mood-to-visual-profile mapping
`dashboard/src/aevp/particles.ts`	Particle system with formations
`dashboard/src/aevp/types.ts`	Frontend render state types
`dashboard/src/aevp/identityPresets.ts`	Per-agent visual identity configs
`dashboard/src/aevp/environment.ts`	Environmental context rendering
`dashboard/src/aevp/speechAnalyser.ts`	Speech-driven visual effects
`dashboard/src/aevp/tonalPresence.ts`	Ambient tonal feedback
`dashboard/src/aevp/toolCategories.ts`	Tool classification for particle effects
`dashboard/src/aevp/shaders/`	GLSL vertex and fragment shaders

Web UI

Dashboard

Tasks

Gateway & Ops

Intent & Safety

Security

Overview

Normalized Agent State

Emotional Color Mapping

Mood Visual Profiles (Examples)

Personality Modulation (Phase 5)

WebGL2 Renderer

Pass 1: Ambient Orb

Pass 2: Particle Overlay

Pass 3: Bloom Post-Process

FBO Architecture

State Transitions

WebSocket Events

Tool Activity Colors

Identity Presets

Gestures

Tool Categories

Speech Analysis

Speech Reactivity

Dashboard Climate

Tonal Presence (Accessibility)

Reduced Motion

Implementation Status

Dashboard Integration

Performance

Configuration

Key Files

Web UI

Dashboard

Tasks

Gateway & Ops

Intent & Safety

Security

​Overview

​Normalized Agent State

​Emotional Color Mapping

​Mood Visual Profiles (Examples)

​Personality Modulation (Phase 5)

​WebGL2 Renderer

​Pass 1: Ambient Orb

​Pass 2: Particle Overlay

​Pass 3: Bloom Post-Process

​FBO Architecture

​State Transitions

​WebSocket Events

​Tool Activity Colors

​Identity Presets

​Gestures

​Tool Categories

​Speech Analysis

​Speech Reactivity

​Dashboard Climate

​Tonal Presence (Accessibility)

​Reduced Motion

​Implementation Status

​Dashboard Integration

​Performance

​Configuration

​Key Files

Overview

Normalized Agent State

Emotional Color Mapping

Mood Visual Profiles (Examples)

Personality Modulation (Phase 5)

WebGL2 Renderer

Pass 1: Ambient Orb

Pass 2: Particle Overlay

Pass 3: Bloom Post-Process

FBO Architecture

State Transitions

WebSocket Events

Tool Activity Colors

Identity Presets

Gestures

Tool Categories

Speech Analysis

Speech Reactivity

Dashboard Climate

Tonal Presence (Accessibility)

Reduced Motion

Implementation Status

Dashboard Integration

Performance

Configuration

Key Files