Azure-AI_Foundry

Lesson 1: Add Your Data to Azure AI Foundry

Overview

Azure AI Foundry provides a streamlined “Add your data” experience that handles the entire ingestion pipeline — from uploading documents to creating a searchable vector index. This lesson walks through configuring data sources, chunking strategies, and embedding models.

📖 Reference: Add data for RAG in Azure AI Foundry

Supported Data Sources

Azure AI Foundry can ingest data from multiple sources:

Source	Description	Best For
File Upload	Upload files directly in the portal	Quick prototyping, small datasets
Azure Blob Storage	Connect to a storage container	Large document collections
Azure AI Search	Use an existing search index	Pre-indexed enterprise data
URLs	Crawl web pages	Public documentation, knowledge bases
Azure SQL / Cosmos DB	Connect to structured databases	Tabular or semi-structured data

The Data Ingestion Pipeline

When you add data, Azure AI Foundry executes this pipeline automatically:

Upload  →  Parse  →  Chunk  →  Embed  →  Index
  │          │         │         │         │
  │        Extract   Split     Convert   Store in
  │        text      into      chunks    Azure AI
  │        from      passages  to        Search
  │        documents            vectors

Step-by-Step in the Portal

Open your project in Azure AI Foundry.
Navigate to Data → + Add data source.
Select your data source type and configure the connection.
Choose your chunking strategy and embedding model.
Review and start the ingestion job.

Chunking Strategies

Chunking determines how documents are split into passages for retrieval. The right strategy directly impacts RAG quality.

Fixed-Size Chunking

Splits text into chunks of a fixed token count with optional overlap.

# Example: Configuring fixed-size chunking
chunking_config = {
    "strategy": "fixed_size",
    "size": 1024,        # tokens per chunk
    "overlap": 128       # overlapping tokens between chunks
}

Parameter	Recommendation
Chunk size	512–1024 tokens for general documents
Overlap	10–20% of chunk size

Semantic Chunking

Splits text at natural boundaries (paragraphs, sections, headings) to preserve meaning.

# Example: Configuring semantic chunking
chunking_config = {
    "strategy": "semantic",
    "max_chunk_size": 2048,
    "min_chunk_size": 128
}

💡 Tip: Use semantic chunking for documents with complex structure — technical manuals, legal contracts, research papers. Use fixed-size for homogeneous text like transcripts or logs.

Choosing the Right Strategy

Document Type	Recommended Chunking	Why
Technical docs (headings, sections)	Semantic	Preserves logical structure
Chat transcripts	Fixed-size (512 tokens)	Uniform, conversational text
Legal contracts	Semantic	Clause boundaries matter
FAQ lists	Page-level or per-question	Each Q&A is a natural unit
Code repositories	File-level or function-level	Logical code boundaries

Choosing Embedding Models

Embedding models convert text chunks into vector representations for semantic search.

Model	Dimensions	Max Tokens	Best For
`text-embedding-ada-002`	1536	8,191	Legacy workloads
`text-embedding-3-small`	1536	8,191	Cost-effective, general use
`text-embedding-3-large`	3072	8,191	Highest accuracy

from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint="https://<your-resource>.openai.azure.com/",
    api_key="<your-key>",
    api_version="2024-10-21"
)

# Generate embeddings for a text chunk
response = client.embeddings.create(
    model="text-embedding-3-small",
    input="Azure AI Foundry provides tools for building AI applications."
)

embedding_vector = response.data[0].embedding
print(f"Vector dimensions: {len(embedding_vector)}")  # 1536

📖 Reference: Azure OpenAI embeddings

Dimension Reduction

text-embedding-3-small and text-embedding-3-large support dimension reduction via the dimensions parameter — useful for reducing storage costs while maintaining reasonable accuracy:

response = client.embeddings.create(
    model="text-embedding-3-large",
    input="Reduce dimensions for cost savings.",
    dimensions=1024  # Reduce from 3072 to 1024
)

Adding Data with the Python SDK

from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential

# Connect to your AI Foundry project
client = AIProjectClient(
    credential=DefaultAzureCredential(),
    endpoint="https://<your-hub>.api.azureml.ms",
    project_name="<your-project>"
)

# Upload and index files for RAG
# The SDK handles chunking, embedding, and indexing
data_source = client.data.create_or_update(
    name="product-docs",
    path="./documents/",           # Local folder with your files
    description="Product documentation for RAG"
)

print(f"Data asset created: {data_source.name}, version {data_source.version}")

📖 Reference: Azure AI Projects SDK

Best Practices

Clean your data — remove headers, footers, boilerplate, and noise before ingestion.
Use metadata — tag chunks with source filename, page number, date, and category for filtering.
Test with a subset — index 50–100 documents first and validate retrieval quality before scaling.
Monitor index size — large indexes increase search latency and cost.
Version your data — track which documents are in each index version for reproducibility.

Common Pitfalls

Problem	Cause	Solution
Poor retrieval quality	Chunks too large or too small	Experiment with chunk sizes (512–1024)
Missing context	No chunk overlap	Add 10–20% overlap
Irrelevant results	No metadata filtering	Add metadata fields to your index
Slow ingestion	Too many large files at once	Batch uploads, use async processing

Summary

Azure AI Foundry’s “Add your data” feature automates the ingestion pipeline.
Choose chunking strategies based on your document structure.
Select embedding models based on accuracy needs and budget.
Clean and prepare data before indexing for the best results.

Additional Resources

← Overview

Next: Lesson 2 — Azure AI Search Integration →