Agent Evaluation
non-cncf
Tech Writing Pass
Content Lifecycle & Evolution State
1
Initial Seed
Metadata Loaded
2
First Research
Automated Gathering
3
Tech Writing
Content Structured
4
Approval
Editorial Verified
Tech Writing Pass Completed: The technical overview and features have been structured by our autonomous content agents. Final editorial approval is pending.
Overview
A generative AI-powered framework for testing virtual agents.
Key Features
Built-in support for popular AWS services like Amazon Bedrock, Amazon Q Business, and Amazon SageMaker
Orchestrate concurrent, multi-turn conversations with your agent while evaluating its responses
Define hooks to perform additional tasks such as integration testing
CI/CD pipeline integration to expedite time to delivery and maintain stability in production
Use Cases
Testing and validating virtual LLM agents with multi-turn conversations before deploying them to production environments.
Getting Started
Visit the documentation to install the package and configure built-in or custom targets for testing.
Recent Updates
Release of v0.4.1 on March 31, 2025.
Did You Know?
Internally implements an LLM evaluator agent to interact with and critique user-provided target agents.