Azure-AI_Foundry

Module 8: Evaluation and Monitoring for AI Applications

Official docs: Evaluate your Generative AI application

Why Evaluation Matters

Deploying an AI application without systematic evaluation is like shipping software without tests. Models can hallucinate, drift, produce unsafe content, or silently degrade in quality. A rigorous evaluation strategy catches these issues before users do.

Key risks without evaluation:

Risk Impact Mitigation
Hallucination Users receive fabricated facts Groundedness evaluators
Relevance drift Answers stop matching user intent Relevance + coherence metrics
Safety violations Harmful or biased outputs Safety evaluators + red-teaming
Performance degradation Latency spikes, cost overruns Production monitoring + alerts

Evaluation Lifecycle

┌─────────────┐     ┌─────────────────┐     ┌──────────────────────┐
│  DEVELOPMENT │────▶│    STAGING       │────▶│    PRODUCTION         │
│              │     │                 │     │                      │
│ • Unit evals │     │ • Batch evals   │     │ • Real-time monitor  │
│ • Prompt     │     │ • Safety scans  │     │ • Quality sampling   │
│   iteration  │     │ • Red-teaming   │     │ • Drift detection    │
│ • Quick      │     │ • CI/CD quality │     │ • Alert & dashboard  │
│   feedback   │     │   gates         │     │ • Cost tracking      │
└─────────────┘     └─────────────────┘     └──────────────────────┘
       │                     │                        │
       ▼                     ▼                        ▼
   azure-ai-evaluation   Automated pipelines    Azure Monitor +
   SDK (local runs)      + Evaluation wizard    App Insights

Built-in Evaluators

Quality Evaluators

Evaluator What It Measures Score Range Requires
GroundednessEvaluator Are claims supported by the context? 1–5 query, response, context
GroundednessProEvaluator Production-grade groundedness (Azure AI service) 1–5 query, response, context
RelevanceEvaluator Does the response address the query? 1–5 query, response, context
CoherenceEvaluator Is the response logically consistent? 1–5 query, response
FluencyEvaluator Is the language natural and readable? 1–5 query, response
SimilarityEvaluator How close is the response to a ground truth? 1–5 query, response, ground_truth
F1ScoreEvaluator Token-level overlap with ground truth 0.0–1.0 response, ground_truth

Safety Evaluators

Evaluator What It Detects Score
ViolenceEvaluator Violent content Severity 0–7
SexualEvaluator Sexual content Severity 0–7
SelfHarmEvaluator Self-harm content Severity 0–7
HateUnfairnessEvaluator Hate speech and bias Severity 0–7
IndirectAttackEvaluator Prompt injection via context (XPIA) Boolean
ProtectedMaterialEvaluator Copyrighted content Boolean

Monitoring Signals in Production

Signal Tool Purpose
Latency (P50 / P95 / P99) Azure Monitor + App Insights Detect slow responses
Token usage (prompt + completion) Custom metrics / OpenTelemetry Cost tracking and budgeting
Error rate (4xx / 5xx) App Insights Availability monitoring
Groundedness score (sampled) Periodic evaluation job Quality drift detection
Safety flags Content Safety API Real-time safety gate
User feedback (thumbs up/down) Custom events Direct quality signal
Trace completeness OpenTelemetry spans Pipeline health

Quick Start: Your First Evaluation

Install the SDK

pip install azure-ai-evaluation azure-identity

Run a Single Evaluation

"""Quick evaluation example using azure-ai-evaluation SDK."""

from azure.ai.evaluation import GroundednessEvaluator, RelevanceEvaluator
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()

# Azure AI project for model-assisted evaluators
model_config = {
    "azure_deployment": "gpt-4o",
    "azure_endpoint": "https://<your-endpoint>.openai.azure.com/",
    "api_version": "2024-12-01-preview",
}

# Initialize evaluators
groundedness_eval = GroundednessEvaluator(model_config=model_config)
relevance_eval = RelevanceEvaluator(model_config=model_config)

# Sample data
query = "What are the benefits of Azure AI Foundry?"
context = (
    "Azure AI Foundry provides a unified platform for building, evaluating, "
    "and deploying AI applications. It offers built-in evaluation tools, "
    "model catalog access, and enterprise-grade security."
)
response = (
    "Azure AI Foundry helps you build and deploy AI apps with built-in "
    "evaluation, access to many models, and enterprise security features."
)

# Run evaluations
groundedness_result = groundedness_eval(
    query=query, context=context, response=response
)
relevance_result = relevance_eval(
    query=query, context=context, response=response
)

print(f"Groundedness: {groundedness_result}")
print(f"Relevance:    {relevance_result}")

Module Roadmap

Lesson Topic Key Skills
Lesson 1 Built-in Evaluators Quality + safety evaluators, batch evaluation, Evaluation wizard
Lesson 2 Custom Evaluators Code-based evaluators, LLM-as-judge, adversarial simulation
Lesson 3 Production Monitoring Azure Monitor, OpenTelemetry, dashboards, alerts
Knowledge Check Scenario-based Quiz 15 questions covering all lessons

Prerequisites

Tip: Model-assisted evaluators (groundedness, relevance, coherence, fluency) require an LLM deployment. Token-overlap metrics like F1ScoreEvaluator run locally with no model needed.

Next → Lesson 1: Built-in Evaluators