Evaluation, testing and release management

If you can't test it, you can't scale it. Agents behave like distributed systems: multiple steps, external tool dependencies, partial failures and non-determinism. They need an AgentOps discipline closer to SRE than to prototyping.

Evaluate across four dimensions

• Task success: end-to-end success rate, completion time and rework rate, not just "seems helpful".
• Policy compliance: forbidden actions avoided, sensitive data not exposed, approvals used correctly.
• Factuality and provenance: citations where needed, no hallucinated sources, up-to-date documents used.
• Tool correctness: right tool, right parameters, right sequence; safe retries and idempotency.

Release management that works

• Unit tests for tools and connectors.
• Scenario tests derived from SOPs.
• A regression suite of golden tasks.
• Canary deployments with automatic rollback triggers.
• Red-teaming: prompt injection, exfiltration and jailbreak attempts.

If you cannot measure agent quality, you cannot claim it's improving.