Azure-AI_Foundry

Module 8: Knowledge Check

Test your understanding of evaluation, monitoring, and safety testing for AI applications. Each question is scenario-based — pick the best answer, then expand to check.

Question 1: Choosing the Right Evaluator

You are building a RAG application that answers questions using company documentation. During testing, you notice the model sometimes includes information not found in the retrieved documents. Which evaluator should you use to detect this?

Answer **C) `GroundednessEvaluator`** Groundedness measures whether the response's claims are supported by the provided context. If the model fabricates information not in the retrieved documents, the groundedness score will be low. `RelevanceEvaluator` checks if the response addresses the query, not whether it's grounded in the context.

Question 2: GroundednessProEvaluator

Your team wants the most accurate groundedness evaluation for production use. What is the difference between GroundednessEvaluator and GroundednessProEvaluator?

Answer **B) Pro uses an Azure AI service for higher accuracy; standard uses a prompted LLM** `GroundednessProEvaluator` uses a dedicated Azure AI Content Safety-based service specifically trained for groundedness detection, providing higher accuracy than the standard evaluator which relies on prompting a general-purpose LLM (like GPT-4o) to judge groundedness.

Question 3: LLM-as-Judge Pattern

You need to evaluate whether your chatbot’s responses match your company’s brand voice (friendly, professional, uses “you” to address users). No built-in evaluator covers this. What approach should you take?

Answer **C) Build an LLM-as-judge evaluator with a brand voice scoring prompt** Brand voice is a subjective, multi-dimensional quality assessment — exactly the kind of evaluation where the LLM-as-judge pattern excels. You define a scoring rubric in the prompt (tone, language style, direct address) and have GPT-4o score each response. A simple keyword check (option B) would miss nuance like tone and overall feel.

Question 4: IndirectAttackEvaluator

Your RAG application retrieves documents from a shared knowledge base that multiple teams contribute to. A security review flags a concern about prompt injection via retrieved content. Which evaluator specifically targets this threat?

Answer **C) `IndirectAttackEvaluator`** `IndirectAttackEvaluator` (also known as XPIA — Cross-domain Prompt Injection Attack) specifically detects prompt injection attacks hidden in the retrieved context. This is critical for RAG applications where the retrieval corpus could be poisoned with adversarial instructions embedded in documents.

Question 5: Batch Evaluation Column Mapping

You have a JSONL dataset with columns question, retrieved_context, answer, and expected_answer. You want to run GroundednessEvaluator. Which column mapping is correct?

Answer **B) `{"query": "${data.question}", "context": "${data.retrieved_context}", "response": "${data.answer}"}`** `GroundednessEvaluator` requires `query`, `context`, and `response` as its input parameters. You map your dataset columns to these using the `${data.}` syntax. The context is what was retrieved, and the response is the model's answer. Option C uses `ground_truth`, which is for `SimilarityEvaluator` or `F1ScoreEvaluator`, not groundedness. </details> --- ### Question 6: OpenTelemetry Traces You've instrumented your RAG pipeline with OpenTelemetry and see this trace in Application Insights: ``` rag_pipeline (2.3s) ├── retrieve_documents (0.8s) ├── rerank_documents (0.2s) └── generate_response (1.3s) ``` Users report slow responses. Based on this trace, where should you focus optimization? - A) Retrieval — it takes the most time after generation - B) Reranking — it's the bottleneck - C) Generation — it accounts for 57% of total latency - D) Add caching at the pipeline level — all steps are slow
Answer **C) Generation — it accounts for 57% of total latency** The `generate_response` span takes 1.3s out of 2.3s total (57%). This is the dominant latency contributor. Optimization strategies include: using a faster model (e.g., `gpt-4o-mini`), reducing prompt length (fewer retrieved documents), enabling streaming, or implementing response caching for common queries.
--- ### Question 7: Adversarial Simulation Before deploying your customer service chatbot, you want to test its resistance to adversarial attacks. You run `AdversarialSimulator` with `AdversarialScenario.ADVERSARIAL_QA`. What does this test? - A) Whether the chatbot gives accurate answers to normal questions - B) Whether the chatbot can be tricked into generating harmful or unsafe responses through adversarial inputs - C) Whether the chatbot's retrieval system returns relevant documents - D) Whether the chatbot handles high traffic loads
Answer **B) Whether the chatbot can be tricked into generating harmful or unsafe responses through adversarial inputs** `AdversarialSimulator` generates adversarial inputs designed to trick the model into violating safety guardrails — generating harmful content, leaking system prompts, or bypassing content filters. It simulates real-world attack patterns that malicious users might attempt. It does NOT test accuracy, retrieval quality, or load handling.
--- ### Question 8: Safety Severity Scores Your safety evaluation returns these results for a response: ``` violence: severity=1, sexual: severity=0, self_harm: severity=0, hate: severity=3 ``` Your quality gate threshold is `severity ≤ 2`. What should happen? - A) Pass — all scores are low - B) Fail — the hate score (3) exceeds the threshold (2) - C) Pass — the average severity (1.0) is below the threshold - D) Fail — any non-zero score should block deployment
Answer **B) Fail — the hate score (3) exceeds the threshold (2)** Quality gates check each safety category independently. The hate/unfairness score of 3 exceeds the threshold of 2, so the gate should fail. You don't average safety scores — a single high-severity category is enough to block. Option D is too strict; severity 1 (low) for violence is generally acceptable.
--- ### Question 9: Cost Monitoring You're tracking token usage with OpenTelemetry custom metrics. Your dashboard shows prompt tokens increasing by 40% week over week, but request volume is flat. What's the most likely cause? - A) Users are sending longer queries - B) The retrieval system is returning more context documents - C) The model is generating longer responses - D) A and B are both plausible — investigate with trace attributes
Answer **D) A and B are both plausible — investigate with trace attributes** Prompt tokens include both the user's query and any context injected into the prompt. The increase could be from longer user queries, more retrieved documents being stuffed into the prompt, or changes to the system prompt. Check your `retrieve.doc_count` and `user.query_length` span attributes to pinpoint the cause. Completion tokens (not prompt tokens) would indicate longer responses, ruling out option C.
--- ### Question 10: Quality Drift Detection Your periodic production evaluation shows groundedness dropping from 4.3 to 3.8 over two weeks. Request volume and user queries haven't changed significantly. What should you investigate first? - A) Whether the LLM model was updated or changed - B) Whether the retrieval index was updated with new or modified documents - C) Whether users are gaming the system with adversarial inputs - D) Both A and B — changes to the model or data are the most likely causes
Answer **D) Both A and B — changes to the model or data are the most likely causes** Quality drift in groundedness typically comes from two sources: (1) the model's behavior changed (new version, fine-tuning update) — it might be relying less on context; or (2) the retrieval data changed (stale documents, broken index, new documents with poor quality). Since user behavior is stable, investigate model and data changes first. Adversarial inputs (C) would more likely affect safety scores than groundedness specifically.
--- ### Question 11: CI/CD Quality Gates Your CI/CD pipeline runs evaluation on every PR that changes prompt templates. The quality gate requires `groundedness >= 4.0` and `relevance >= 4.0`. The latest run shows `groundedness=4.1, relevance=3.8`. What should happen? - A) Pass — groundedness is above threshold and compensates for relevance - B) Fail — relevance (3.8) is below the 4.0 threshold - C) Pass — the average of 3.95 rounds up to 4.0 - D) Manual review — close scores should be reviewed by a human
Answer **B) Fail — relevance (3.8) is below the 4.0 threshold** Quality gates are strict: each metric must independently meet its threshold. There's no averaging or compensation between metrics. The pipeline should block the merge and require the developer to investigate why relevance dropped. Common fixes: adjust the prompt template, improve retrieval, or add more relevant examples to the evaluation dataset.
--- ### Question 12: Auto-Instrumentation Configuration You enable auto-instrumentation for Azure OpenAI SDK calls with `AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED=true`. A compliance review flags this setting. Why? - A) It slows down API calls significantly - B) It records prompt and completion text in traces, which may contain PII - C) It sends telemetry to Microsoft servers without consent - D) It's incompatible with Azure Monitor
Answer **B) It records prompt and completion text in traces, which may contain PII** When content recording is enabled, the full prompt text (including user queries) and completion text are captured in OpenTelemetry traces and stored in Application Insights. If user queries contain personal information (names, emails, medical details), this data ends up in your telemetry store. In production, set this to `false` or omit it entirely, and rely on token counts and latency metrics instead.
--- ### Question 13: Composite Evaluator Design You're building a composite evaluator with these weights: groundedness (30%), relevance (25%), coherence (15%), brand voice (20%), word count (10%). A response scores groundedness=5, relevance=5, coherence=5, brand voice=2, word count=1.0. The composite threshold is 0.7. Does it pass? - A) Yes — most scores are perfect - B) No — the brand voice score drags it below threshold - C) Need to calculate: normalize scores, apply weights, compare to 0.7
Answer **C) Need to calculate: normalize scores, apply weights, compare to 0.7** Calculation (quality scores 1–5 normalized to 0–1): - Groundedness: (5-1)/4 = 1.0 × 0.30 = 0.300 - Relevance: (5-1)/4 = 1.0 × 0.25 = 0.250 - Coherence: (5-1)/4 = 1.0 × 0.15 = 0.150 - Brand voice: (2-1)/4 = 0.25 × 0.20 = 0.050 - Word count: 1.0 × 0.10 = 0.100 **Total: 0.850** → ✅ Passes (0.850 ≥ 0.7) Despite the low brand voice score, the strong quality scores pull the composite above threshold. However, you should still flag the brand voice issue for improvement.
--- ### Question 14: KQL Query for Token Anomaly You want to write a KQL query that alerts when hourly token consumption exceeds **2× the 7-day moving average**. Which approach is correct? - A) Use `summarize` with `avg()` over 7 days and compare to the current hour - B) Use `make-series` with `series_decompose_anomalies()` for automatic detection - C) Use `percentile(value, 99)` to find outliers - D) Both A and B would work, but B is more robust for anomaly detection
Answer **D) Both A and B would work, but B is more robust for anomaly detection** Option A (manual threshold) works for simple cases: ```kql customMetrics | where name == "ai.tokens.total" | summarize HourlyTokens = sum(value) by bin(timestamp, 1h) | extend AvgTokens = avg_over_7d(...) -- simplified | where HourlyTokens > 2 * AvgTokens ``` Option B (`series_decompose_anomalies`) is more robust because it accounts for seasonal patterns (e.g., lower usage on weekends, peaks at 10 AM) and doesn't require you to manually set the 2× multiplier. It's the recommended approach for production anomaly detection in Azure Monitor.
--- ### Question 15: End-to-End Monitoring Strategy You're launching a new AI-powered customer support application. Which combination of monitoring components provides the most comprehensive coverage? - A) Application Insights only — it covers traces, metrics, and logs - B) OpenTelemetry tracing + custom token metrics + KQL alerts + periodic quality evaluation - C) Quality evaluation in CI/CD only — if tests pass, production will be fine - D) User feedback collection only — users will report problems
Answer **B) OpenTelemetry tracing + custom token metrics + KQL alerts + periodic quality evaluation** A comprehensive monitoring strategy needs multiple layers: 1. **OpenTelemetry tracing** — see the full request lifecycle (retrieval → generation → safety check) 2. **Custom metrics** — track tokens, latency, and costs in real-time 3. **KQL alerts** — proactively detect anomalies (latency spikes, error surges, budget overruns) 4. **Periodic quality evaluation** — sample production traffic and run evaluators to detect quality drift Option A provides infrastructure but misses AI-specific quality monitoring. Option C misses production drift entirely. Option D is reactive, not proactive. You need all four layers working together.
--- ## Scoring Guide | Score | Level | Recommendation | |---|---|---| | 13–15 correct | ⭐ Expert | You're ready to build production AI evaluation and monitoring | | 10–12 correct | ✅ Proficient | Review the lessons for topics you missed | | 7–9 correct | 📖 Developing | Re-read Lessons 1–3 and try the code examples | | 0–6 correct | 🔄 Beginning | Start with the Overview and work through each lesson hands-on | --- [← Lesson 3: Production Monitoring](/Azure-AI_Foundry/course/module-08-evaluation-and-monitoring/lesson-3-production-monitoring.html) | [Back to Overview](/Azure-AI_Foundry/course/module-08-evaluation-and-monitoring/overview.html)