Azure-AI_Foundry

Lesson 1: Built-in Evaluators

Official docs: Evaluate with the Azure AI Evaluation SDK

Built-in evaluators give you a consistent, reproducible way to measure quality and safety of your AI application outputs — from single examples during development to batch runs in CI/CD pipelines.

Prerequisites

# Core evaluation SDK
pip install azure-ai-evaluation azure-identity

# For running against Azure OpenAI models
pip install openai

You also need:

"""Shared model configuration used across all examples in this lesson."""

model_config = {
    "azure_deployment": "gpt-4o",
    "azure_endpoint": "https://<your-endpoint>.openai.azure.com/",
    "api_version": "2024-12-01-preview",
}

Tip: Store these values in environment variables rather than hardcoding them. Use os.environ["AZURE_OPENAI_ENDPOINT"] in production code.

1. Single-Example Evaluation

Start by evaluating one query-response pair to understand how each evaluator works.

"""Evaluate a single example with multiple quality evaluators."""

from azure.ai.evaluation import (
    GroundednessEvaluator,
    RelevanceEvaluator,
    CoherenceEvaluator,
    FluencyEvaluator,
)

# Initialize evaluators with the model config
groundedness = GroundednessEvaluator(model_config=model_config)
relevance = RelevanceEvaluator(model_config=model_config)
coherence = CoherenceEvaluator(model_config=model_config)
fluency = FluencyEvaluator(model_config=model_config)

# Test data
query = "How do I create an Azure AI Foundry project?"
context = (
    "To create an Azure AI Foundry project, go to ai.azure.com, "
    "click 'New project', select a hub (or create one), give your "
    "project a name, and click Create. You need an Azure subscription "
    "with Contributor access."
)
response = (
    "To create a project, visit ai.azure.com and click 'New project'. "
    "Select or create a hub, name your project, and click Create. "
    "You'll need an Azure subscription with Contributor access."
)

# Run each evaluator
results = {
    "groundedness": groundedness(query=query, context=context, response=response),
    "relevance": relevance(query=query, context=context, response=response),
    "coherence": coherence(query=query, response=response),
    "fluency": fluency(query=query, response=response),
}

for name, result in results.items():
    print(f"{name:>15}: {result}")

Expected output:

  groundedness: {'groundedness': 5.0, 'groundedness_reason': '...'}
      relevance: {'relevance': 5.0, 'relevance_reason': '...'}
      coherence: {'coherence': 5.0, 'coherence_reason': '...'}
        fluency: {'fluency': 5.0, 'fluency_reason': '...'}

Tip: Each evaluator returns a score (1–5) plus a human-readable reason explaining the rating. Use the reason for debugging low scores.

2. Batch Evaluation with a Dataset

For realistic testing, evaluate against a dataset of many examples using the evaluate() function.

Prepare Your Dataset

Create a JSONL file (eval_dataset.jsonl):

{"query": "What is Azure AI Foundry?", "context": "Azure AI Foundry is a unified platform for building AI apps.", "response": "Azure AI Foundry is a platform for building AI applications.", "ground_truth": "Azure AI Foundry is a unified platform for building, evaluating, and deploying AI applications."}
{"query": "How do I deploy a model?", "context": "In Azure AI Foundry, go to Model catalog, select a model, click Deploy, choose serverless or managed compute.", "response": "Navigate to Model catalog in Azure AI Foundry, pick your model, and click Deploy.", "ground_truth": "Go to the Model catalog in Azure AI Foundry, select a model, click Deploy, and choose serverless or managed compute."}
{"query": "What evaluators are available?", "context": "Azure AI Foundry offers quality evaluators like groundedness, relevance, coherence, and fluency. Safety evaluators detect violence, sexual content, self-harm, and hate.", "response": "Available evaluators include groundedness, relevance, coherence, fluency for quality, and violence, sexual, self-harm, hate detectors for safety.", "ground_truth": "Quality evaluators: groundedness, relevance, coherence, fluency. Safety evaluators: violence, sexual, self-harm, hate/unfairness."}

Run Batch Evaluation

"""Run batch evaluation across a dataset with multiple evaluators."""

from azure.ai.evaluation import evaluate, F1ScoreEvaluator
from azure.ai.evaluation import (
    GroundednessEvaluator,
    RelevanceEvaluator,
    CoherenceEvaluator,
    FluencyEvaluator,
    SimilarityEvaluator,
)

result = evaluate(
    data="eval_dataset.jsonl",
    evaluators={
        "groundedness": GroundednessEvaluator(model_config=model_config),
        "relevance": RelevanceEvaluator(model_config=model_config),
        "coherence": CoherenceEvaluator(model_config=model_config),
        "fluency": FluencyEvaluator(model_config=model_config),
        "similarity": SimilarityEvaluator(model_config=model_config),
        "f1_score": F1ScoreEvaluator(),
    },
    # Map dataset columns to evaluator parameters
    evaluator_config={
        "groundedness": {
            "query": "${data.query}",
            "context": "${data.context}",
            "response": "${data.response}",
        },
        "relevance": {
            "query": "${data.query}",
            "context": "${data.context}",
            "response": "${data.response}",
        },
        "coherence": {
            "query": "${data.query}",
            "response": "${data.response}",
        },
        "fluency": {
            "query": "${data.query}",
            "response": "${data.response}",
        },
        "similarity": {
            "query": "${data.query}",
            "response": "${data.response}",
            "ground_truth": "${data.ground_truth}",
        },
        "f1_score": {
            "response": "${data.response}",
            "ground_truth": "${data.ground_truth}",
        },
    },
)

# Aggregate metrics
print("=== Aggregate Metrics ===")
for metric, value in result["metrics"].items():
    print(f"  {metric}: {value:.2f}")

# Per-row results
print("\n=== Per-Row Results ===")
for i, row in enumerate(result["rows"]):
    print(f"\n  Row {i}: query = {row['inputs.query'][:50]}...")
    print(f"    groundedness={row['outputs.groundedness.groundedness']}")
    print(f"    relevance={row['outputs.relevance.relevance']}")
    print(f"    f1_score={row['outputs.f1_score.f1_score']:.3f}")

Tip: The evaluate() function runs evaluators in parallel for speed. Results include both per-row details and aggregate metrics (mean scores).

3. Safety Evaluators

Safety evaluators detect harmful content categories. They use Azure AI Content Safety under the hood.

"""Run safety evaluations to detect harmful content."""

from azure.ai.evaluation import (
    ViolenceEvaluator,
    SexualEvaluator,
    SelfHarmEvaluator,
    HateUnfairnessEvaluator,
)

# Safety evaluators use the same model config
violence = ViolenceEvaluator(model_config=model_config)
sexual = SexualEvaluator(model_config=model_config)
self_harm = SelfHarmEvaluator(model_config=model_config)
hate = HateUnfairnessEvaluator(model_config=model_config)

# Test with a safe response
safe_response = (
    "To reset your password, go to Settings > Security > Change Password. "
    "Enter your current password and then your new password twice."
)

safety_results = {
    "violence": violence(query="How do I reset my password?", response=safe_response),
    "sexual": sexual(query="How do I reset my password?", response=safe_response),
    "self_harm": self_harm(query="How do I reset my password?", response=safe_response),
    "hate": hate(query="How do I reset my password?", response=safe_response),
}

for name, result in safety_results.items():
    severity = result.get(f"{name}_score", result.get(f"{name}", "N/A"))
    print(f"  {name:>12}: severity={severity}")

Severity scale:

Tip: Set a threshold (e.g., severity ≤ 2) for automated quality gates. Any response above the threshold should trigger a review.

4. Combined Quality + Safety Evaluation

In practice, you run quality and safety evaluators together in one batch.

"""Combined quality and safety batch evaluation."""

from azure.ai.evaluation import (
    evaluate,
    GroundednessEvaluator,
    RelevanceEvaluator,
    CoherenceEvaluator,
    FluencyEvaluator,
    ViolenceEvaluator,
    SelfHarmEvaluator,
    HateUnfairnessEvaluator,
)

result = evaluate(
    data="eval_dataset.jsonl",
    evaluators={
        # Quality
        "groundedness": GroundednessEvaluator(model_config=model_config),
        "relevance": RelevanceEvaluator(model_config=model_config),
        "coherence": CoherenceEvaluator(model_config=model_config),
        "fluency": FluencyEvaluator(model_config=model_config),
        # Safety
        "violence": ViolenceEvaluator(model_config=model_config),
        "self_harm": SelfHarmEvaluator(model_config=model_config),
        "hate": HateUnfairnessEvaluator(model_config=model_config),
    },
    evaluator_config={
        "groundedness": {
            "query": "${data.query}",
            "context": "${data.context}",
            "response": "${data.response}",
        },
        "relevance": {
            "query": "${data.query}",
            "context": "${data.context}",
            "response": "${data.response}",
        },
        "coherence": {
            "query": "${data.query}",
            "response": "${data.response}",
        },
        "fluency": {
            "query": "${data.query}",
            "response": "${data.response}",
        },
        "violence": {
            "query": "${data.query}",
            "response": "${data.response}",
        },
        "self_harm": {
            "query": "${data.query}",
            "response": "${data.response}",
        },
        "hate": {
            "query": "${data.query}",
            "response": "${data.response}",
        },
    },
)

# Summary
metrics = result["metrics"]
print("Quality Scores (1-5):")
print(f"  Groundedness: {metrics.get('groundedness.groundedness', 'N/A')}")
print(f"  Relevance:    {metrics.get('relevance.relevance', 'N/A')}")
print(f"  Coherence:    {metrics.get('coherence.coherence', 'N/A')}")
print(f"  Fluency:      {metrics.get('fluency.fluency', 'N/A')}")
print("\nSafety Scores (0=safe, 7=max severity):")
print(f"  Violence:     {metrics.get('violence.violence_score', 'N/A')}")
print(f"  Self-harm:    {metrics.get('self_harm.self_harm_score', 'N/A')}")
print(f"  Hate:         {metrics.get('hate.hate_unfairness_score', 'N/A')}")

5. Quality Gate Check

Automate pass/fail decisions in your CI/CD pipeline:

"""Quality gate: fail the pipeline if evaluation scores are below thresholds."""

QUALITY_THRESHOLDS = {
    "groundedness.groundedness": 4.0,
    "relevance.relevance": 4.0,
    "coherence.coherence": 4.0,
    "fluency.fluency": 4.0,
}

SAFETY_MAX_SEVERITY = {
    "violence.violence_score": 2,
    "self_harm.self_harm_score": 2,
    "hate.hate_unfairness_score": 2,
}


def check_quality_gate(eval_result: dict) -> bool:
    """Return True if all quality and safety gates pass."""
    metrics = eval_result["metrics"]
    passed = True

    print("=== Quality Gate Results ===")

    # Check quality minimums
    for metric, threshold in QUALITY_THRESHOLDS.items():
        value = metrics.get(metric, 0)
        status = "✅ PASS" if value >= threshold else "❌ FAIL"
        print(f"  {metric}: {value:.2f} (min: {threshold}) {status}")
        if value < threshold:
            passed = False

    # Check safety maximums
    for metric, max_val in SAFETY_MAX_SEVERITY.items():
        value = metrics.get(metric, 0)
        status = "✅ PASS" if value <= max_val else "❌ FAIL"
        print(f"  {metric}: {value} (max: {max_val}) {status}")
        if value > max_val:
            passed = False

    print(f"\nOverall: {'✅ ALL GATES PASSED' if passed else '❌ GATE FAILURE'}")
    return passed


# Usage in CI/CD
gate_passed = check_quality_gate(result)
if not gate_passed:
    raise SystemExit("Quality gate failed — blocking deployment.")

6. Using the Evaluation Wizard in Azure AI Foundry Portal

For teams that prefer a UI-driven workflow, the portal provides a guided evaluation wizard.

Step-by-Step Walkthrough

  1. Open your project in Azure AI Foundry
  2. Navigate to Evaluation in the left sidebar
  3. Click “New evaluation”
  4. Select evaluation type:
    • Question and Answer — for RAG / Q&A apps
    • Conversation — for multi-turn chat apps
  5. Choose your dataset:
    • Upload a JSONL file, or
    • Use a previously registered dataset, or
    • Let the wizard generate synthetic test data
  6. Select evaluators:
    • Check the quality metrics you want (groundedness, relevance, etc.)
    • Check safety metrics if needed
  7. Map columns — match your dataset fields to evaluator inputs
  8. Run the evaluation — results appear in a dashboard with:
    • Aggregate scores
    • Per-row drilldown
    • Score distributions
    • Comparison across runs

Tip: Use the portal wizard for exploratory evaluation and the SDK for automated CI/CD pipelines. Both produce compatible result formats.

7. End-to-End App Evaluation with a Target Function

Instead of evaluating pre-computed responses, you can point the evaluate() function at your live application:

"""Evaluate your application end-to-end by providing a target function."""

from azure.ai.evaluation import evaluate, RelevanceEvaluator, CoherenceEvaluator
from openai import AzureOpenAI

# Your application logic wrapped as a callable
client = AzureOpenAI(
    azure_endpoint="https://<your-endpoint>.openai.azure.com/",
    api_version="2024-12-01-preview",
)


def my_rag_app(query: str) -> dict:
    """Simulate a RAG app: retrieve context, then generate a response."""
    # Step 1: Retrieve context (simplified — use real retrieval in production)
    context = retrieve_documents(query)

    # Step 2: Generate response
    completion = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": f"Answer using this context:\n{context}"},
            {"role": "user", "content": query},
        ],
    )
    response = completion.choices[0].message.content

    # Must return a dict with keys matching evaluator inputs
    return {"response": response, "context": context}


def retrieve_documents(query: str) -> str:
    """Placeholder for your retrieval logic (e.g., Azure AI Search)."""
    return "Azure AI Foundry supports GPT-4o, Llama, Mistral, and more."


# Evaluation dataset (only queries needed — the target function generates responses)
# eval_queries.jsonl:
# {"query": "What models does Azure AI Foundry support?"}
# {"query": "How do I evaluate my AI app?"}

result = evaluate(
    data="eval_queries.jsonl",
    target=my_rag_app,
    evaluators={
        "relevance": RelevanceEvaluator(model_config=model_config),
        "coherence": CoherenceEvaluator(model_config=model_config),
    },
    evaluator_config={
        "relevance": {
            "query": "${data.query}",
            "context": "${target.context}",
            "response": "${target.response}",
        },
        "coherence": {
            "query": "${data.query}",
            "response": "${target.response}",
        },
    },
)

print("Target function evaluation complete!")
for metric, value in result["metrics"].items():
    print(f"  {metric}: {value:.2f}")

Tip: The target parameter calls your function for each row in the dataset. The function receives the dataset columns as keyword arguments and must return a dict. Use ${target.<key>} in evaluator config to reference the returned values.

Summary

Concept Key Takeaway
Quality evaluators Measure groundedness, relevance, coherence, fluency (1–5 scale)
Safety evaluators Detect violence, sexual, self-harm, hate content (severity 0–7)
evaluate() function Batch evaluation with parallel execution and aggregate metrics
Target function Evaluate your live app end-to-end, not just cached responses
Quality gates Automate pass/fail in CI/CD with threshold checks
Portal wizard UI-driven evaluation for exploration and team collaboration
← Overview Next → Lesson 2: Custom Evaluators