AI Observability and Debugging
Part of: Effective AI Utilization — Table of Contents
AI calls are black boxes. The input goes in, the output comes out, and when something goes wrong, you need instrumentation to understand why. BrianBot has minimal observability — token counts and pass/fail status. Here's what production AI systems need.
What BrianBot Tracks Today
Per-episode: total input tokens, total output tokens, processing status (PENDING/PROCESSING/COMPLETED/FAILED), timestamps. Per-run: UsageRecord with token counts and a zero-value cost field. This tells you that something happened and roughly how much it cost, but not why it succeeded or failed.
What's Missing
Prompt logging: Store the actual prompts sent and responses received, at least for failed calls. When output quality degrades, you need to diff the prompt against a known-good version.
Latency tracking: How long does each AI call take? Is Sonnet getting slower? Is Haiku consistently under 2 seconds? Latency trends surface provider issues before they become outages.
Quality signals: Beyond pass/fail, is the output good? For structured outputs (topic extraction, metadata), you can validate format and completeness programmatically. For generative outputs, track human feedback or downstream metrics (email open rates for companion content).
Error classification: Not all failures are equal. A rate limit error (retryable) is different from a content policy violation (not retryable) is different from a malformed response (maybe retryable with a different prompt). Classify errors to inform your retry and fallback logic (see Model Fallback and Resilience).
The Observability Stack
Minimum viable: structured logging with prompt/response pairs, latency, model used, token counts, and error details. Write to your existing logging infrastructure. Next level: dedicated AI observability tools (Langfuse, Helicone, Braintrust) that provide dashboards, cost analysis, prompt versioning, and quality evaluation.
Debug Mode
For development, a "verbose mode" that logs full prompt/response pairs is invaluable. For production, store a hash of the prompt and the full response for the last N calls per step, rotating to manage storage.
Related: Model Fallback and Resilience, Cost Tracking and Budget Controls, AI Pipeline Design, Token Optimization Playbook