Amazon Bedrock Model Evaluation Practical Guide - Automatic Metrics, LLM-as-a-Judge, RAG Evaluation, and CI/CD Quality Gates

First Published: 2026-07-14
Last Updated: 2026-07-14

Every team that ships a generative AI feature eventually reaches the same crossroads. A prompt is rewritten, a newer foundation model is swapped in, a knowledge base is re-chunked — and someone declares the output "better." It usually is, for the handful of examples they happened to look at. But "better on the examples I checked" is not a release criterion, and it does not survive contact with a deployment pipeline. A pipeline needs a number: a score, a threshold, a pass or fail. This article is about producing that number with Amazon Bedrock's built-in evaluation feature, and about wiring it into the point in your delivery process where a regression can still be stopped.

Amazon Bedrock Evaluations is a managed capability for scoring the quality of models and Retrieval-Augmented Generation (RAG) systems. It runs the evaluation as a job: you hand it a dataset of prompts in Amazon S3, it produces responses (or scores responses you supply), and it writes back computed metrics. There is no evaluation harness to build, no judge model to host, no scoring code to maintain. That managed shape is exactly what makes it a good fit for a quality gate — a gate is only trustworthy if it is cheap and repeatable enough to run on every change.

The scope of this article is deliberately narrow: the evaluation feature itself. It covers the job types, how to shape a dataset, the exact built-in metrics and which task each one fits, the prerequisites (IAM, S3, CORS, encryption) that trip up a first job, and the mechanics of gating a deployment on an evaluation job. What it does not cover is the surrounding architecture. The broader LLMOps picture — observability, tracing, drift detection, online evaluation of live traffic, and the reference architecture that ties them together — is a topic of its own, covered in LLMOps Observability and Evaluation Architecture on AWS. Evaluating agents — trajectories, tool selection, multi-step behavior — is a different product (Amazon Bedrock AgentCore Evaluations) with its own guide, Amazon Bedrock AgentCore Evaluations Practical Guide. This article stays one layer below both: the evaluation job as an implementation primitive. Terminology used throughout — evaluator model, generator, faithfulness, grounding — is defined in the Amazon Bedrock Glossary.

One framing note before the mechanics. Amazon Bedrock Evaluations splits into two families that share a job model but answer different questions. Model evaluation asks "how good are this model's responses?" RAG evaluation asks "how good is this retrieval-and-generation system at answering from my data?" The two families overlap in their metrics and their dataset formats, but they are configured, scoped, and interpreted differently, and most of this article treats them in turn.

1. Evaluation Job Types at a Glance

Everything in Amazon Bedrock Evaluations is created through one API operation, CreateEvaluationJob, and a single top-level field decides which family you are in. applicationType takes exactly two values: ModelEvaluation or RagEvaluation. Within each family there are distinct job types, distinguished by how the scoring is performed.

Amazon Bedrock Evaluations job types and their inputs and outputs

Model evaluation offers three job types:

Programmatic (automatic) evaluation scores responses with traditional natural-language algorithms — no judge model involved. Metrics such as BERTScore, F1, and exact-match variants are computed against curated built-in datasets or a custom dataset you supply. This is the deterministic, reference-based path.
LLM-as-a-Judge (model-as-judge) evaluation uses a second foundation model — the evaluator (judge) model — to score the responses of the model under test (the generator). The judge assigns each response a score and an explanation. This is now generally available, and it is the path for nuanced quality dimensions (correctness, completeness, faithfulness) that a simple string metric cannot capture.
Human-based evaluation routes responses to a work team of human reviewers for subjective rating. The team can be your own employees or an AWS-managed workforce, and it is orchestrated through Amazon SageMaker Ground Truth work teams.

RAG evaluation offers two job types, both powered by an LLM-as-a-judge behind the scenes:

Retrieve only evaluates the retrieval stage in isolation — how relevant and complete the retrieved passages are — against an Amazon Bedrock Knowledge Base or your own retrieval pipeline.
Retrieve and generate evaluates the whole pipeline end to end — retrieval plus the generated answer — using a knowledge base and a response-generator model, or your own responses.

The following table is the decision surface. Pick the row that matches the question you are actually asking; the rest of the article implements each one.

* You can sort the table by clicking on the column name.

Job type	Family	How it scores	Needs ground truth?	Best for
Programmatic	Model	Algorithmic metrics (BERTScore, F1, accuracy, toxicity)	Yes, for accuracy/robustness/QA	Deterministic, reproducible scoring at scale; reference-based tasks
LLM-as-a-Judge	Model	Evaluator model scores + explains	Optional (used for correctness/completeness)	Nuanced quality without exhaustive ground truth
Human-based	Model	Human reviewers rate	Optional (shown to workers)	Subjective judgment; a baseline to calibrate automated scores
Retrieve only	RAG	LLM-as-a-judge on retrieved passages	Needed for context coverage	Diagnosing the retrieval half of a RAG system
Retrieve and generate	RAG	LLM-as-a-judge on end-to-end answers	Optional	Scoring the full RAG answer, grounding included

Two API-shape facts are worth carrying forward from the start. First, evaluationConfig is a union — a job is either automated (which covers both programmatic and LLM-as-a-Judge) or human, never both. Second, inferenceConfig describes what is evaluated: for an automated model evaluation job it holds a single model (or inference profile), while a human evaluation job supports two models or inference profiles so reviewers can compare them side by side. When you bring your own inference responses instead of asking Bedrock to invoke a model, the inference step is skipped and the job scores the responses you supplied.

2. Preparing Datasets and Prerequisites

This is where first jobs fail, so it is worth being exact. A dataset is a file in Amazon S3 in JSON Lines format (.jsonl extension), where each line is one valid JSON object. Every evaluation job — model or RAG, automatic or human — accepts up to 1,000 prompts in a dataset. For an automatic model evaluation job, all data used and generated must live in an S3 bucket in the same AWS Region as the job.

The exact keys depend on the job type. Getting them wrong is the most common cause of a job that creates but produces no useful scores.

2.1 Model evaluation dataset (programmatic and human)

A programmatic dataset uses three keys. prompt is required and holds the input — the text to generate from, the question to answer, the passage to summarize, or the text to classify. referenceResponse is the ground-truth answer and is required for the accuracy and robustness metrics and for the question-and-answer task type; it is what the response is scored against. category is optional and, when present, produces per-category score breakdowns in the report.

{"prompt": "Bobigny is the capital of", "referenceResponse": "Seine-Saint-Denis", "category": "Capitals"}
{"prompt": "Aurillac is the capital of", "referenceResponse": "Cantal", "category": "Capitals"}

A human-based dataset uses the same three keys; the difference is that prompt and referenceResponse are shown to the human reviewers in the worker UI, so referenceResponse here is a reference the worker consults rather than a string a metric is computed against.

2.2 Model-as-judge with bring-your-own-inference (BYOI)

If you have already generated responses — from a Bedrock model, a fine-tuned model, or a system hosted anywhere — you can score them without asking Bedrock to invoke anything. Add a modelResponses array. A model-as-judge job supports exactly one model response per prompt, and every row must use the same modelIdentifier.

{"prompt": "The prompt used to generate the response", "referenceResponse": "(optional) ground truth", "category": "(optional)", "modelResponses": [{"response": "The response your model generated", "modelIdentifier": "my-fine-tuned-model-v3"}]}

If you supply referenceResponse, the judge uses it when computing the Builtin.Correctness and Builtin.Completeness metrics; both metrics also work without a ground truth, using different judge prompts.

2.3 RAG evaluation dataset

A RAG dataset is shaped around conversation turns. The top-level object holds a conversationTurns array; each turn has a prompt, and optionally referenceResponses (the expected answer), referenceContexts (the expected retrieved passages, so you can compare actual retrieval to expected), and an output block when you bring your own retrieval or generation results.

{
  "conversationTurns": [
    {
      "prompt": { "content": [{ "text": "What is the refund window for enterprise plans?" }] },
      "referenceResponses": [{ "content": [{ "text": "Enterprise plans have a 30-day refund window." }] }],
      "referenceContexts": [{ "content": [{ "text": "Section 4.2: Enterprise refunds are processed within 30 days of purchase." }] }]
    }
  ]
}

For retrieve-and-generate jobs the maximum number of turns per conversation is 5; for retrieve-only jobs you can specify only a single turn. referenceContexts is what lets a retrieve-only job measure context coverage against the passages you expected to see.

When you bring your own RAG results rather than pointing at a knowledge base, each turn carries an output block: the generated text, the knowledgeBaseIdentifier or modelIdentifier that produced it, and a retrievedPassages.retrievalResults array of the passages your pipeline actually retrieved (each with its content.text and optional metadata). That is what lets Bedrock score an external RAG system without invoking anything — it evaluates the retrieved passages and the generated answer you supply against the reference fields.

2.4 Prerequisites: IAM, S3, CORS, and encryption

Beyond the dataset, four setup items gate a first job.

Model access. The evaluator model and the generator model (or the response-generator model for RAG) must be enabled on the Model access page of the Amazon Bedrock console in the Region you run in. A job that references a model you have not been granted access to fails.

IAM service role. CreateEvaluationJob runs under a service role that Bedrock assumes. Its trust policy must name bedrock.amazonaws.com as the service principal. The permissions it needs depend on the job type:

Automatic (programmatic and judge): read access to the S3 bucket holding a custom prompt dataset, write access to the S3 bucket where results are saved, and bedrock:InvokeModel on the models the job invokes (the model under test and, for judge jobs, the evaluator model).
Human-based: the S3 permissions above plus the additional permissions for the Amazon SageMaker Ground Truth integration that manages the work team, including the Amazon Augmented AI (Amazon A2I) human-loop actions (sagemaker:StartHumanLoop, sagemaker:DescribeFlowDefinition, and related actions) that let Amazon Bedrock create the review resources on your behalf.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadDatasetWriteResults",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::my-eval-bucket",
        "arn:aws:s3:::my-eval-bucket/*"
      ]
    },
    {
      "Sid": "InvokeGeneratorAndEvaluatorModels",
      "Effect": "Allow",
      "Action": ["bedrock:InvokeModel"],
      "Resource": [
        "arn:aws:bedrock:us-east-1::foundation-model/<generator-model-id>",
        "arn:aws:bedrock:us-east-1::foundation-model/<evaluator-model-id>"
      ]
    }
  ]
}

The trust policy is the piece people forget:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "bedrock.amazonaws.com" },
      "Action": "sts:AssumeRole"
    }
  ]
}

CORS — the non-obvious one. It is easy to assume every evaluation job needs a CORS configuration on its S3 bucket, and some getting-started material reads that way. The current Amazon Bedrock Developer Guide is more precise: CORS is required only for human-based evaluation jobs, on the S3 output bucket, and is not required for automatic model evaluation or RAG evaluation jobs. The reason is the human-review UI, which reads from and writes to that bucket from the browser. If you are gating a pipeline with automatic jobs, you do not need a CORS configuration at all; if you add human review, configure CORS on the output bucket first, or the worker UI cannot load the data.

Encryption. During a job, Bedrock creates a temporary copy of your evaluation data and, by default, encrypts it with an AWS managed key. For tighter control, pass a customer managed key ARN in the customerEncryptionKeyId field of CreateEvaluationJob and grant the service role the necessary AWS KMS permissions. The temporary copy is deleted when the job completes.

3. Automatic (Programmatic) Evaluation and Metric Selection

Programmatic evaluation is the deterministic path: no judge, just algorithms. Its central design decision is choosing the task type, because the task type dictates which metrics are available and which built-in datasets apply. You choose exactly one task type per job.

There are four task types, and each pairs specific metrics with specific computed values. Applying a metric to the wrong task is the classic mistake — BERTScore is meaningful for summarization, not for a binary classifier — so the mapping below is the part to get right.

* You can sort the table by clicking on the column name.

Task type	Metric	Computed value
General text generation	Accuracy	Real World Knowledge (RWK) score
General text generation	Robustness	Word error rate (under semantic-preserving perturbations)
General text generation	Toxicity	Toxicity score
Text summarization	Accuracy	BERTScore
Text summarization	Robustness	BERTScore and deltaBERTScore
Text summarization	Toxicity	Toxicity score
Question and answer	Accuracy	NLP-F1
Question and answer	Robustness	F1 and deltaF1
Question and answer	Toxicity	Toxicity score
Text classification	Accuracy	Classification accuracy (binary accuracy)
Text classification	Robustness	Classification accuracy and its delta

A few notes on how these are computed, because the names hide the mechanics. Accuracy for general text generation is the RWK score, which probes how well the model encodes factual knowledge about the real world; higher is better. For summarization it is BERTScore, a semantic-similarity measure between the generated summary and the reference; for question-and-answer it is NLP-F1, the token-overlap F1 between answer and reference; for classification it is straightforward binary accuracy. Robustness measures how much the output changes under minor, meaning-preserving perturbations of the input — Bedrock perturbs prompts by lowercasing, injecting keyboard typos, converting numbers to words, randomly changing case, and adding or removing whitespace, then measures the resulting drift (via word error rate, or the delta of the task's accuracy metric). A low robustness drift is the desirable outcome. Toxicity scores harmful content in the output. The delta variants (deltaBERTScore, deltaF1) express this drift directly as the drop in the task's accuracy metric between the clean prompt and its perturbed form, so a small delta means the model held its answer under noise. In the API, these three are the only valid metricNames for an automated (programmatic) job: Builtin.Accuracy, Builtin.Robustness, and Builtin.Toxicity — the Builtin.Correctness-family identifiers in Section 4 belong to judge-based jobs and are rejected here.

You can run programmatic jobs against a curated built-in dataset (for example, TREX and WikiText2 for general text generation, Gigaword for summarization, BoolQ, NaturalQuestions, and TriviaQA for question-and-answer, and Women's E-Commerce Clothing Reviews for classification) or against a custom dataset you supply. Built-in datasets are the fastest way to get a comparative baseline across models; a custom dataset is how you measure on prompts that resemble your actual traffic. The tradeoff is coverage versus fidelity: use a built-in set to shortlist candidate models on a standard corpus, then use a custom set drawn from your own traffic to make the final call on the inputs you actually serve.

When you read the report, note that only responses the model actually produced are used in the metric calculation. If the report shows fewer responses than prompts, some prompts errored at inference time — check the data output file in your S3 bucket rather than trusting the summary number. Programmatic metrics are also not perfectly deterministic run to run, because the underlying models are nondeterministic; expect small variation on repeated runs of the same dataset.

4. LLM-as-a-Judge: Evaluator Models, Metrics, and Honest Interpretation

Programmatic metrics answer "does the response match the reference," which is exactly wrong for open-ended generation where there is no single correct string. LLM-as-a-Judge fills that gap: a second model reads the response and scores it on quality dimensions, with an explanation for each score. It is generally available and integrates directly with the rest of the evaluation feature.

A judge job has two models. The generator produces the responses (or you supply them via BYOI), and the evaluator — the judge — scores them. Bedrock provides a curated set of judge models: pre-selected foundation models paired with tuned evaluation prompts, maintained and updated by the Bedrock team. You do not bring an external judge model; you select one from the curated set. Because the curated list changes as models are added, this article does not reproduce it — confirm the current judge-model options in the console or the Developer Guide rather than hardcoding a list that will age.

4.1 The built-in metrics

The LLM-as-a-Judge metrics group into four categories. The exact identifiers matter when you specify them in the API, so they are given as Builtin.* names:

Quality — Builtin.Correctness, Builtin.Completeness, Builtin.Faithfulness (faithfulness is hallucination detection: does the response contain information not present in the prompt or context).
User experience — Builtin.Helpfulness, Builtin.Coherence (logical coherence), Builtin.Relevance.
Instruction compliance — Builtin.FollowingInstructions, Builtin.ProfessionalStyleAndTone.
Safety / responsible AI — Builtin.Harmfulness, Builtin.Stereotyping, Builtin.Refusal.

Builtin.Correctness and Builtin.Completeness use the ground-truth referenceResponse when you supply one, and fall back to a reference-free judge prompt when you do not.

4.2 Creating a judge job

The following creates a model-as-judge job from the CLI. Note the shape: evaluationConfig.automated carries both the datasetMetricConfigs (task type, dataset location, and the metricNames to compute) and the evaluatorModelConfig.bedrockEvaluatorModels that names the judge; inferenceConfig.models names the generator; outputDataConfig.s3Uri is where results land.

{
  "jobName": "prompt-v7-vs-baseline",
  "roleArn": "arn:aws:iam::<account-id>:role/BedrockEvaluationRole",
  "applicationType": "ModelEvaluation",
  "evaluationConfig": {
    "automated": {
      "datasetMetricConfigs": [
        {
          "taskType": "General",
          "dataset": {
            "name": "prod_like_prompts",
            "datasetLocation": { "s3Uri": "s3://my-eval-bucket/datasets/prod_like.jsonl" }
          },
          "metricNames": ["Builtin.Correctness", "Builtin.Completeness", "Builtin.Faithfulness", "Builtin.Harmfulness"]
        }
      ],
      "evaluatorModelConfig": {
        "bedrockEvaluatorModels": [{ "modelIdentifier": "<evaluator-model-id>" }]
      }
    }
  },
  "inferenceConfig": {
    "models": [{ "bedrockModel": { "modelIdentifier": "<generator-model-id>" } }]
  },
  "outputDataConfig": { "s3Uri": "s3://my-eval-bucket/results/" }
}

aws bedrock create-evaluation-job --cli-input-json file://judge_job.json

To score responses you generated yourself, keep inferenceConfig — it is a required field — but replace bedrockModel with precomputedInferenceSource, setting its inferenceSourceIdentifier to match the modelIdentifier in your dataset's modelResponses BYOI shape shown earlier; Bedrock skips the invoke step and scores your data directly.

4.3 Custom metrics

When the built-in metrics do not capture what you care about — adherence to a brand voice, a domain-specific rubric, a categorical classification of responses — you can define custom metrics. Custom metrics extend the same LLM-as-a-Judge framework: you write a judge prompt describing the criterion and the scoring scale (numerical or categorical), and the evaluator applies it alongside or instead of the built-in metrics. Custom metrics are available for both model and RAG evaluation.

4.4 Reading the scores, and not over-reading them

Every score is normalized to a value between 0 and 1. For quality and user-experience metrics, closer to 1 is better. For the safety metrics the direction inverts: a high Builtin.Harmfulness, Builtin.Stereotyping, or Builtin.Refusal is a worse result, not a better one — a knowledge base can score a healthy 0.82 on completeness and an alarming 0.94 on stereotyping in the same report. Read each metric in its own direction.

Three cautions keep judge scores honest. First, the judge is a model, so its scores are nondeterministic — the same dataset can yield slightly different numbers on re-runs, which matters when you set a gate threshold (leave margin; do not gate on a delta smaller than the run-to-run noise). Second, judges have a self-preference bias: a model tends to rate outputs from its own family more highly. AWS's guidance is to choose an evaluator from a different model family than the generator, which is the single most effective mitigation. Third — the honesty point that applies to every safety metric — a response passing a Builtin.Harmfulness check is evidence, not proof. "Cleared the judge" is not the same as "safe"; the judge is one layer in a defense-in-depth posture, alongside guardrails, human review of the tail, and monitoring, not a substitute for them.

4.5 Reading the judge output

The judge does not only emit a number. For each response it returns a score and a short explanation of why it scored that way, and both are written to the S3 output alongside the aggregate report card shown in the console. The aggregate is what you gate on; the per-response explanations are what you triage with. When a metric comes back low, sort the output by that metric, read the explanations on the worst records, and the failures usually cluster — a category of prompt the model mishandles, a systematic hallucination, a formatting regression — far faster than re-reading raw responses would reveal. Treat the console report card as the summary and the S3 output object as the evidence.

5. RAG Evaluation: Separating Retrieval Quality from Generation Quality

A RAG system can fail in two independent places. The retriever can hand over passages that do not contain the answer, or the generator can mishandle passages that do. If you only measure the end-to-end answer, you cannot tell which half is broken. Amazon Bedrock's RAG evaluation gives you two job types precisely so you can measure them separately.

Both types are driven by an LLM-as-a-judge under the hood, and both can target an Amazon Bedrock Knowledge Base directly or accept bring-your-own-inference responses from an external RAG pipeline. The job's applicationType is RagEvaluation.

5.1 Retrieve only

A retrieve-only job scores the retrieval stage in isolation, using two metrics:

Builtin.ContextRelevance — how contextually relevant the retrieved passages are to the question.
Builtin.ContextCoverage — how much of the information in the ground-truth passages the retrieved passages cover. This metric requires a ground truth, which you supply as referenceContexts in the dataset.

This is the job to run when you are tuning chunking, hybrid search, metadata filtering, or reranking and you want a number for retrieval that is not muddied by generation. The retrieval-quality tuning itself — which knob fixes which symptom — is a subject of its own, covered in Amazon Bedrock Knowledge Bases Retrieval Quality Engineering; RAG evaluation is how you measure whether a change to those knobs helped.

5.2 Retrieve and generate

A retrieve-and-generate job scores the full pipeline — retrieval plus the generated answer — with a richer metric set:

Generation quality: Builtin.Correctness, Builtin.Completeness, Builtin.Helpfulness, Builtin.LogicalCoherence.
Grounding: Builtin.Faithfulness (does the answer avoid hallucinating relative to the retrieved passages), Builtin.CitationPrecision (were the cited passages cited correctly), Builtin.CitationCoverage (is the answer well supported by its citations, with none missing).
Responsible AI: Builtin.Harmfulness, Builtin.Stereotyping, Builtin.Refusal.

One naming subtlety is worth flagging because it will bite anyone who copies metric names between job types: the logical-coherence metric is Builtin.Coherence in a model-as-judge job but Builtin.LogicalCoherence in a RAG retrieve-and-generate job. The concepts are the same; the identifiers differ. Copy the metric name from the doc for the job type you are actually creating.

The grounding metrics are what make retrieve-and-generate valuable beyond a plain model evaluation. Builtin.Faithfulness catches the answer that sounds right but is not supported by the retrieved context; the citation metrics catch the answer that cites the wrong passage or fails to cite a passage it relied on. Together they turn "the answer looked good" into "the answer is grounded in the retrieved evidence," which is the property a RAG system exists to provide.

6. Human Evaluation

Some quality dimensions resist automation. Whether an answer aligns with a brand voice, whether a summary is appropriate for a regulated audience, whether two models' outputs are meaningfully different in a way a rubric cannot capture — these call for human judgment. Human evaluation is Amazon Bedrock's job type for that.

You assemble a work team, managed through Amazon SageMaker Ground Truth, of either your own employees or an AWS-managed workforce of skilled evaluators; with the AWS-managed option, AWS hires and manages the reviewers on your behalf. You bring your own dataset and can define custom metrics — relevance, style, alignment to brand voice — beyond any built-in ones. Bedrock handles task distribution, scoring, and result aggregation, and a "workers per prompt" parameter controls how many independent ratings each prompt-response pair receives. A human evaluation job can also compare two models side by side, which is why inferenceConfig for a human job accepts two models where an automated job accepts one.

One lifecycle caveat before you build on this job type: behind the scenes, a human-based evaluation job is implemented on top of Amazon Augmented AI (Amazon A2I) resources — the Amazon Bedrock service role must allow the service to create and manage the underlying human loops (sagemaker:StartHumanLoop and related actions). Amazon A2I stops accepting new customer onboarding on July 30, 2026, so confirm current availability for your account before designing a new workflow around human-based evaluation jobs.

As with the automated path, a human job can either have Bedrock invoke the models for you — you provide only prompts, and Bedrock generates the responses your reviewers rate — or you can bring responses you generated elsewhere and have the team rate those. Results come back as ratings aggregated across your reviewers per metric, read in the same report-card view as the automated jobs.

Human evaluation earns its cost in two situations. The first is any dimension genuinely beyond a rubric — the subjective and the regulated. The second is calibration: a small human-rated set is the baseline against which you validate that your automated judge scores actually track human preference, so you can then run the automated judge at scale with confidence. Remember the CORS prerequisite here specifically — human jobs need CORS on the S3 output bucket because the worker UI reads from it in the browser. The end-to-end design of human-in-the-loop review workflows (routing, confidence thresholds, escalation) is a broader architecture question and is out of scope for this feature-level reference.

7. CI/CD Quality Gates

An evaluation that nobody acts on is a dashboard. The point of scoring a model or a prompt change is to make a regression unable to ship. That means putting the evaluation job inside the delivery pipeline as a gate: on a model swap or a prompt change, run an evaluation job, wait for it, read the metrics, and fail the build when a metric drops below a floor or regresses against the current baseline.

CI/CD quality gate driven by a managed Amazon Bedrock evaluation job

The managed-job approach is one of two ways to build such a gate, and it is worth being explicit about the difference. You can build a self-managed gate — a harness in a build step that calls a judge model directly, aggregates scores in your own code, and emits a JUnit report; that pattern, wired through CodeBuild and a golden dataset, is covered in the CI/CD section of LLMOps Observability and Evaluation Architecture on AWS. This section covers the other option: gating on a managed Bedrock evaluation job, where Bedrock owns the scoring and your pipeline owns only the orchestration and the threshold decision. The managed path trades some flexibility for far less code to maintain, and the evaluation job's S3 output is itself a build artifact you can archive.

The gate has four steps: start, poll, read, decide.

Start. The pipeline calls CreateEvaluationJob with the candidate configuration — the new model ID, or the new prompt baked into the dataset — pointing at your golden dataset in S3. Pass a clientRequestToken for idempotency so a retried pipeline step does not launch a duplicate job, and tag the job so you can trace it back to the commit.

Poll. CreateEvaluationJob is asynchronous. Poll GetEvaluationJob until status leaves InProgress. The terminal values are Completed, Failed, Stopping, Stopped, and Deleting; treat anything other than Completed as a gate failure and surface the job's failureMessages.

Read. On Completed, the metric scores are written to the S3 location you named in outputDataConfig. Read the result object, extract the per-metric scores, and compare each against your policy.

Decide. Compare each score to an absolute floor and, if you keep a stored baseline, to the baseline delta. If any monitored metric falls below its floor or regresses more than the allowed margin, exit non-zero so the pipeline stage fails and the change is held. Otherwise, promote.

import json
import time
import boto3

bedrock = boto3.client("bedrock")
s3 = boto3.client("s3")

# 1. Start the evaluation job for the candidate configuration.
job = bedrock.create_evaluation_job(
    jobName=f"gate-{commit_sha}",
    roleArn=EVAL_ROLE_ARN,
    clientRequestToken=commit_sha,            # idempotent per commit
    applicationType="ModelEvaluation",
    evaluationConfig={
        "automated": {
            "datasetMetricConfigs": [{
                "taskType": "General",
                "dataset": {"name": "golden", "datasetLocation": {"s3Uri": GOLDEN_S3_URI}},
                "metricNames": ["Builtin.Correctness", "Builtin.Completeness", "Builtin.Faithfulness"],
            }],
            "evaluatorModelConfig": {"bedrockEvaluatorModels": [{"modelIdentifier": EVALUATOR_ID}]},
        }
    },
    inferenceConfig={"models": [{"bedrockModel": {"modelIdentifier": CANDIDATE_ID}}]},
    outputDataConfig={"s3Uri": RESULTS_S3_URI},
)
job_arn = job["jobArn"]

# 2. Poll until the job leaves InProgress.
while True:
    status = bedrock.get_evaluation_job(jobIdentifier=job_arn)["status"]
    if status != "InProgress":
        break
    time.sleep(30)

if status != "Completed":
    raise SystemExit(f"Evaluation job did not complete: {status}")

# 3. Read the per-metric scores from the S3 output and 4. gate on thresholds.
scores = load_metric_scores_from_s3(s3, RESULTS_S3_URI, job_arn)   # your parser over the output object
FLOORS = {"Builtin.Correctness": 0.85, "Builtin.Completeness": 0.80, "Builtin.Faithfulness": 0.90}
failures = [m for m, floor in FLOORS.items() if scores.get(m, 0.0) < floor]
if failures:
    raise SystemExit(f"Quality gate failed on: {failures} -> blocking deploy")
print("Quality gate passed")

An evaluation job is asynchronous and takes real wall-clock time — the model under test is invoked over every prompt and the judge scores every response — so give the pipeline stage a generous timeout and cap the poll loop with a maximum wall-clock deadline rather than looping forever. A few operational conveniences make the gate robust: ListEvaluationJobs lets a re-run detect an in-flight job for the same commit instead of launching a duplicate; StopEvaluationJob cancels one that a superseding commit has made irrelevant; and because the job's S3 output is a durable object, archive it keyed by commit so a later investigation can diff scores across releases. When a gate passes and the change ships, promote its scores to the stored baseline so the next change is measured against the new bar rather than a stale one — a rolling baseline is what keeps "no regression" meaningful as quality improves.

Multilingual consistency as a regression check. If your application serves multiple languages, a change that improves the aggregate score can still regress one locale badly — the aggregate hides it. Keep one golden dataset per language, run the gate on each, and gate on the minimum across languages rather than the mean. A model swap that lifts English by two points while dropping Japanese below its floor should fail the gate; averaging would let it through.

What this section deliberately stops short of is the deployment mechanics that follow a passed gate — canary traffic shifting between model versions, automated rollback on a post-deploy signal, and the rest of a safe rollout. Those belong to a rollout strategy, covered in Safe Foundation Model Rollout Strategies on AWS; here the gate's job ends at "pass or block."

8. Diagnostics: When a Job Fails or a Score Misleads

Evaluation jobs fail in a small number of characteristic ways, and the fix is usually in the setup, not the data.

The job fails to create or fails immediately. Read failureMessages from GetEvaluationJob first. The usual culprits are IAM and setup: the service role cannot read the dataset object or write to the results prefix; the trust policy does not name bedrock.amazonaws.com; the role lacks bedrock:InvokeModel on the generator or evaluator model; model access was never granted for one of those models on the Model access page; or, for an automatic job, the dataset and results buckets are in a different Region than the job.

The dataset is rejected or scored oddly. A line that is not valid JSON, more than 1,000 prompts, the wrong keys for the job type (for example, a modelResponses array on a job that expects Bedrock to invoke a model, or a missing referenceResponse where a metric requires ground truth), or — for retrieve-and-generate — more than five turns in a conversation. Validate the JSONL locally before you upload.

The human job's worker UI will not load. This is the CORS case. Human-based jobs require a CORS configuration on the S3 output bucket; automatic and RAG jobs do not. If a human job creates fine but reviewers see an empty or broken UI, check the bucket's CORS configuration.

Fewer responses than prompts in the report. Some prompts errored at inference and produced no response; only successful responses enter the metric calculation. Open the data output file in S3 to see which prompts failed and why, rather than treating the smaller denominator as normal.

The scores move between identical runs. Both the models under test and the judge are nondeterministic, so repeated runs of the same dataset produce slightly different numbers. This is expected. The implication for gating is to set thresholds with margin and to avoid failing a build on a delta smaller than the observed run-to-run variance; if you need tighter signal, increase the dataset size so the aggregate is more stable.

The judge seems to flatter one model. Self-preference bias — the judge rating its own family higher — is the most common way a judge score misleads. Use an evaluator from a different model family than the generator, and, for a high-stakes decision, corroborate the judge against a small human-rated baseline before trusting it at scale.

9. Frequently Asked Questions

Is Amazon Bedrock model evaluation the same as AgentCore Evaluations?
No. Amazon Bedrock Evaluations scores model responses and RAG systems. Amazon Bedrock AgentCore Evaluations is a separate product for evaluating agents — trajectories, tool use, and multi-step behavior. Use this feature for models and RAG; use AgentCore Evaluations for agents.

Do I have to let Bedrock invoke the model, or can I score responses I already have?
Both. If you provide only prompts, Bedrock invokes the model for you. If you provide a modelResponses array (model-as-judge) or an output block (RAG), Bedrock skips the invoke step and scores your bring-your-own-inference data — which is also how you evaluate a non-Bedrock model.

Which is better, programmatic or LLM-as-a-Judge?
They answer different questions. Programmatic metrics (BERTScore, F1, accuracy) are deterministic and reference-based — ideal when you have ground truth and want reproducibility. LLM-as-a-Judge scores nuanced quality (correctness, completeness, faithfulness) without needing an exact reference string. Many teams run both.

How do I keep a judge from just preferring its own outputs?
Choose an evaluator model from a different model family than the model being evaluated. Self-preference bias is real, and cross-family judging is the primary mitigation. For decisions that matter, calibrate against a small human-rated set.

Does the automatic evaluation path require CORS on my S3 bucket?
No. Per the Amazon Bedrock Developer Guide, CORS is required only for human-based evaluation jobs, on the S3 output bucket. Automatic model evaluation and RAG evaluation jobs do not require a CORS configuration.

What is the difference between the retrieve-only and retrieve-and-generate RAG jobs?
Retrieve-only scores the retrieval stage alone (context relevance and, against ground-truth passages, context coverage). Retrieve-and-generate scores the whole pipeline, including generation quality, faithfulness, and citation precision and coverage. Run retrieve-only to isolate a retrieval problem; run retrieve-and-generate to judge the end-to-end answer.

How many prompts can a dataset hold?
Up to 1,000 prompts per dataset per evaluation job. For retrieve-and-generate RAG jobs, each conversation can have at most 5 turns; retrieve-only jobs allow a single turn.

Can I evaluate a model that is not hosted on Amazon Bedrock?
Yes, through bring-your-own-inference. Generate the responses with your own model or system, put them in the dataset (a modelResponses array for model-as-judge, or an output block for RAG), and the job scores your responses without invoking anything on Bedrock. The judge and the metrics are the same; only the source of the responses differs.

10. Summary

Amazon Bedrock Evaluations turns "the output feels better" into a score a pipeline can enforce, and it does so as a managed job so the scoring is cheap and repeatable enough to gate on. Everything runs through CreateEvaluationJob, split by applicationType into model evaluation and RAG evaluation. Model evaluation offers three job types — programmatic (deterministic, reference-based metrics like BERTScore and F1, chosen by task type), LLM-as-a-Judge (a curated evaluator model scoring nuanced quality with the Builtin.* metrics), and human-based (a SageMaker Ground Truth work team for subjective judgment and judge calibration). RAG evaluation offers two — retrieve-only, which isolates retrieval with context relevance and coverage, and retrieve-and-generate, which scores the whole pipeline including faithfulness and citation metrics. The prerequisites that trip up a first job are concrete and worth internalizing: a JSONL dataset of at most 1,000 prompts in same-Region S3, a service role trusting bedrock.amazonaws.com with S3 read/write and model-invoke permissions, model access granted for both the generator and the evaluator, and CORS only for human jobs. Wired into CI/CD, an evaluation job becomes a gate — create, poll GetEvaluationJob to Completed, read the S3 scores, and block the deploy when a metric falls below its floor, gating on the minimum across languages to catch locale regressions. Read every score in its own direction, leave margin for judge nondeterminism, and treat a passed safety metric as evidence rather than proof. For the surrounding architecture and rollout mechanics, follow the delegated guides referenced throughout.

11. References

References:
Tech Blog with curated related content

Written by Hidekazu Konishi