feat: add RAGAS adapter for third-party eval metrics

[brandonaxu]

Summary

Adds RAGASAdapter to the third-party evaluation adapters, alongside the existing DeepEvalAdapter and AutoEvalsAdapter. Wraps any RAGAS metric as an AgentCore code-based Lambda evaluator with the same BaseAdapter pattern.

Design

Scores metrics through RAGAS's per-sample APIs (metric.single_turn_score() / metric.multi_turn_score() for legacy metrics, metric.score(**kwargs) for ragas.metrics.collections metrics) rather than the batch ragas.evaluate() pipeline. This means the adapter itself adds no dependency on datasets/pyarrow/pandas, which makes it compatible with slim ragas builds for size-constrained Lambda deployments.

Three-way dispatch:

• Legacy single-turn (Faithfulness, ExactMatch, ContextPrecision, ...): SingleTurnSample + single_turn_score()
• Legacy multi-turn (ToolCallAccuracy, TopicAdherenceScore, AgentGoalAccuracy): MultiTurnSample + multi_turn_score()
• Collections / decorator metrics (ragas.metrics.collections, @discrete_metric, @numeric_metric): metric.score(**fields) with signature-based field filtering

Features

• Validates metric.required_columns before scoring — returns MISSING_REQUIRED_FIELD with actionable guidance instead of letting ragas return silent 0.0 scores
• Parses ground truth and retrieval context embedded in the user message (\n\nReference Answer:\n and \n\nContext:\n separators) since ADOT trace formats have no dedicated fields for these
• Recovers ranked chunk lists from JSON-serialized embedded contexts, preserving chunk boundaries for rank-aware metrics (LLMContextPrecision)
• Handles threshold=None metrics (e.g. SemanticSimilarity) and provides an adapter-level threshold= override for collections metrics which carry none
• Discrete (string-valued) metrics produce categorical labels natively
• Preserves MetricResult.reason as the explanation
• ImportError at score time returns MISSING_DEPENDENCY (not a generic crash)
• Multi-turn messages are stripped of embedded markers so ground truth doesn't leak into the conversation being judged
• RAGASAdapter is the canonical class; RagasAdapter alias provided

Usage

from bedrock_agentcore.evaluation.custom_code_based_evaluators import custom_code_based_evaluator
from bedrock_agentcore.evaluation.custom_code_based_evaluators.third_party.ragas import RAGASAdapter
from ragas.metrics import Faithfulness
from ragas.llms import LangchainLLMWrapper
from langchain_aws import ChatBedrockConverse

eval_llm = LangchainLLMWrapper(ChatBedrockConverse(
    model_id="us.anthropic.claude-sonnet-4-5-20250929-v1:0",
    region_name="us-east-1",
))
adapter = RAGASAdapter(metric=Faithfulness(), llm=eval_llm)

@custom_code_based_evaluator()
def handler(eval_input, context=None):
    return adapter(eval_input, context)

Known Limitations

• reference_contexts (ground-truth contexts for reference-comparison metrics) is not defaulted from retrieved contexts — supply it via custom_mapper when a genuine ground-truth source exists
• reference_tool_calls built from expected_trajectory.toolNames carry names without arguments; arg-accurate comparison needs a custom_mapper with full ToolCall objects
• reference_topics (TopicAdherenceScore) has no span source — requires custom_mapper
• Dual-mode metrics (AspectCritic, SimpleCriteriaScore) always score single-turn
• ragas <1.0 imports datasets when the ragas package is imported; the adapter is compatible with trimmed builds but does not solve that import itself

Dependency Notes

• Pins ragas>=0.4.3,<1.0.0 — the tested version range
• Pins langchain-community>=0.3.0,<0.4.2 — langchain-community 0.4.2 removed chat_models.vertexai, which ragas <1.0 imports unconditionally
• Adds ragas to the evaluation integ test extra-deps in CI
• uv.lock regenerated for the new extra; incidental bumps to langchain-community resolution only

Testing

• 48 unit tests covering all dispatch branches, embedded parsing (plain text + JSON chunk lists), validation, multi-turn messages, threshold override, error codes
• 8 integration tests with real deterministic metrics (legacy ExactMatch, collections ExactMatch, @discrete_metric, ToolCallAccuracy match/mismatch, adapter-imports-without-datasets)
• All pre-commit hooks pass; uv lock --check passes

[jariy17]

YOu have merge conflicts and small nit

[jariy17]

Why can't we use this helper function instead

[brandonaxu]

Good catch — switched to _make_agent_evaluator_input in the latest push. The separate helper was a leftover from earlier development when the shared one used a span format the mappers couldn't parse, but that's been fixed on main since. All ragas integ tests should now use the shared helper.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 83.85417% with 31 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@5d4ca0d). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
...code_based_evaluators/third_party/ragas/adapter.py	83.68%	24 Missing and 7 partials ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #618   +/-   ##
=======================================
  Coverage        ?   88.56%           
=======================================
  Files           ?      120           
  Lines           ?    10023           
  Branches        ?     1527           
=======================================
  Hits            ?     8877           
  Misses          ?      763           
  Partials        ?      383           

Flag	Coverage Δ
unittests	88.56% <83.85%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.