Azure-AI_Foundry

Module 5: RAG & Data Grounding — Overview

Introduction

Retrieval-Augmented Generation (RAG) is the most widely adopted pattern for building AI applications that need to answer questions using your own data. Instead of relying solely on a model’s training data, RAG retrieves relevant information at query time and injects it into the prompt — grounding the model’s response in facts.

📖 Reference: Retrieval-Augmented Generation in Azure AI Foundry

What Is Retrieval-Augmented Generation (RAG)?

RAG combines two capabilities:

Retrieval — searching a knowledge base (documents, databases, APIs) for information relevant to the user’s query.
Generation — passing the retrieved context to a large language model to produce a grounded answer.

┌──────────────┐     ┌──────────────────┐     ┌──────────────────┐
│  User Query  │────▶│  Search / Retrieve│────▶│  LLM Generation  │
│              │     │  (AI Search)      │     │  (Azure OpenAI)  │
└──────────────┘     └──────────────────┘     └──────────────────┘
        │                     │                        │
        │              Relevant chunks           Grounded answer
        │              from your data            with citations
        ▼                     ▼                        ▼

Without RAG, a model can only use what it learned during training — which may be outdated, incomplete, or entirely missing your proprietary data.

Why Grounding Matters

The Hallucination Problem

Large language models generate plausible-sounding text, but they can fabricate facts — a phenomenon called hallucination. Grounding mitigates this by providing the model with verified source material.

Without Grounding	With Grounding (RAG)
Model invents answers	Model cites retrieved documents
No source attribution	Responses include references
Stale training data	Real-time access to current data
Generic answers	Domain-specific, accurate answers

Benefits of RAG

Accuracy — answers are based on your actual documents and data.
Currency — no need to retrain; update your index when data changes.
Cost — far cheaper than fine-tuning for most knowledge-grounding scenarios.
Transparency — you can show users the source documents behind each answer.
Control — enforce document-level security so users only see what they’re authorized to access.

RAG Architecture Pattern

A production RAG system in Azure AI Foundry typically involves these components:

Ingestion Pipeline (Offline)

Documents  →  Chunking  →  Embedding  →  Indexing
 (PDF, DOCX,    (Split into     (Convert to      (Store in
  HTML, etc.)    passages)       vectors)         AI Search)

Query Pipeline (Online)

User Query  →  Embed Query  →  Search Index  →  Build Prompt  →  LLM  →  Response
                                (hybrid search)   (system msg +
                                                   retrieved chunks)

Key Azure Services

Component	Azure Service
LLM	Azure OpenAI Service (GPT-4o, GPT-4.1)
Search & Retrieval	Azure AI Search
Embeddings	Azure OpenAI (text-embedding-3-small/large)
Orchestration	Azure AI Foundry / Prompt Flow
Data Storage	Azure Blob Storage, Azure SQL, Cosmos DB

📖 Reference: Azure AI Search overview

RAG vs Fine-Tuning vs Prompt Engineering

Approach	Best For	Cost	Effort
Prompt Engineering	Formatting, persona, simple instructions	Low	Low
RAG	Domain knowledge, current data, Q&A over docs	Medium	Medium
Fine-Tuning	Style, tone, specialized behavior, structured output	High	High

💡 Tip: Start with RAG before considering fine-tuning. Most enterprise use cases are best served by RAG because it provides accurate, up-to-date answers without the cost and complexity of model training.

What You’ll Learn in This Module

Lesson	Topic
Lesson 1	Adding your data to Azure AI Foundry
Lesson 2	Azure AI Search integration for RAG
Lesson 3	Advanced RAG patterns
Knowledge Check	Test your understanding

Prerequisites

Completed Modules 1–4 (Azure AI Foundry setup, model deployment, prompt engineering)
An Azure subscription with Azure OpenAI and Azure AI Search resources
Python 3.10+ with azure-ai-projects, azure-search-documents, and openai packages

Additional Resources

Next: Lesson 1 — Add Your Data →