AI Reliability Engine (aireliability) — Deterministic Testing & Regression
Deterministic assertion framework, tool-call validation, and regression testing for LLM workflows and autonomous agents.
aireliabilityThis project is authored and published strictly as an in-process Python library on the Python Package Index. It does not require a hosted dashboard or cloud service — executing natively within the consumer's application runtime.
Problem Statement
AI agents are inherently stochastic. When an agent fails, developers struggle to distinguish between normal output variance, tool hallucination, argument corruption, or breaking model updates.
Engineering Goals
- Create deterministic assertions over agent execution properties that must never be stochastic (e.g. tool sequence, schema validity, cost budget).
- Build a developer-friendly CLI (`airel`) that runs in CI/CD alongside pytest.
- Classify agent failures into a standardized failure taxonomy.
- Publish as a clean, dependency-minimal Python package on PyPI.
AI Reliability Engine — System ArchitectureVerified Pipeline
Select any pipeline stage below to inspect verified engineering details and data flow.
Stage: Test Specification
Role: Test DefinitionDeclares expected tool sequence, argument constraints, latency budgets, and cost thresholds.
Execution Pipeline & Mechanics
Deterministic Test Definition
Engineers declare required invariants: e.g. tool 'search_database' must execute before 'send_notification'.
Non-Intrusive Trace Capture
Intercepts runtime execution to capture exact payload arguments, response structures, latency, and tokens.
Contract & Invariant Evaluation
Validates inputs against Pydantic schemas and ensures tool call ordering matches the directed acyclic graph.
Failure Taxonomy & Regression Reporting
Generates clear diffs indicating whether the failure was an argument schema mismatch, sequence violation, or cost overrun.
Engineering Challenges & Design Decisions
Decision: Deterministic trace assertion over LLM-as-a-judge
Rationale: LLM-as-a-judge tests are noisy, expensive, and fail to catch subtle schema violations. Deterministic code assertions provide 100% reproducible test outcomes in CI.
Alternative Rejected: Prompt-based LLM evaluation frameworks.
Core Architectural Highlights
The framework wraps agent runtimes with an execution tracer that records every tool call and model transition. Test suites define declarative assertions using Pydantic models. Traces are validated deterministically, producing rich terminal diffs and CI exit codes.
Testing & Empirical Evaluation
362 automated pytest tests validating trace recorders, argument validator schemas, assertion matchers, and CLI entry points.
Security & Reliability Defenses
Key Engineering Learnings
- •Separating execution tracing from assertion evaluation allowed testing mock and live agent runs with identical test suites.