Azure-AI_Foundry

Azure AI Foundry — Reference Architectures

Production-ready architecture patterns for Azure AI Foundry applications. Each architecture includes a Mermaid diagram, component descriptions, and implementation guidance.

📖 Azure AI Foundry architecture overview

---

1. Basic Chat Application

A simple chat application using Azure AI Foundry with a single model deployment.

graph LR
    subgraph Client
        A[Web / Mobile App]
    end

    subgraph Azure
        subgraph AI_Foundry["Azure AI Foundry"]
            B[API Gateway]
            C[GPT-4o Deployment]
            D[Content Filter]
        end
        E[Azure App Service]
        F[Azure Key Vault]
    end

    A -->|HTTPS| E
    E -->|REST API| B
    B --> D --> C
    E -->|Secrets| F
    C -->|Response| B --> E --> A

Components

Component	Purpose	SKU
Azure App Service	Hosts the web application	B1+
AI Foundry Hub + Project	Manages the AI resources	Standard
GPT-4o Deployment	Chat completion model	Standard PAYG
Content Filter	Input/output safety filtering	Included
Key Vault	Stores API keys and connection strings	Standard

Implementation Notes

• Use DefaultAzureCredential for authentication — no API keys in code.
• Enable content filtering at the deployment level.
• Implement retry logic with exponential backoff for 429 errors.
• Stream responses to reduce time-to-first-token.

📖 Quickstart: Chat with Azure OpenAI models

---

2. Enterprise RAG Pattern

Full retrieval-augmented generation architecture with AI Foundry, AI Search, and Blob Storage.

graph TB
    subgraph Ingestion["Data Ingestion Pipeline"]
        A1[Source Documents] --> A2[Azure Blob Storage]
        A2 --> A3[Document Cracking & Chunking]
        A3 --> A4[Embedding Model]
        A4 --> A5[Azure AI Search Index]
    end

    subgraph Runtime["Runtime Query Flow"]
        B1[User Query] --> B2[App Service / Container App]
        B2 --> B3[Embedding Model]
        B3 --> B4[Azure AI Search]
        B4 -->|Top-K Results| B5[Prompt Assembly]
        B2 --> B5
        B5 --> B6[GPT-4o]
        B6 -->|Content Filter| B7[Response + Citations]
        B7 --> B2 --> B1
    end

    subgraph Governance["Governance & Security"]
        C1[Managed Identity]
        C2[Azure Key Vault]
        C3[Private Endpoints]
        C4[Azure Monitor]
    end

    A5 -.->|Same Index| B4
    C1 -.-> B2
    C1 -.-> B4
    C2 -.-> B2
    C3 -.-> B4
    C4 -.-> B2

Components

Component	Purpose	SKU
Azure Blob Storage	Document storage	Standard LRS
Azure AI Search	Vector + keyword search	Standard S1+
Embedding Model	text-embedding-3-large	Standard PAYG
GPT-4o	Chat completion with grounding	Standard PAYG or PTU
App Service / Container App	Application hosting	B2+ / Consumption+
Managed Identity	Passwordless authentication	System-assigned
Key Vault	Secret management	Standard
Azure Monitor	Logging and metrics	Pay-per-use

Data Flow

1. Ingestion: Documents → Blob Storage → Chunking (512–1024 tokens) → Embeddings → AI Search index
2. Query: User question → Embed query → Hybrid search (vector + keyword + semantic ranking) → Top-K chunks → Assemble prompt → GPT-4o → Response with citations

Key Design Decisions

• Hybrid search (vector + keyword + semantic ranking) for best retrieval quality.
• Chunk overlap of 10–15% to preserve context across boundaries.
• Separate embedding and chat models for independent scaling.
• Citations returned with responses for traceability.

📖 RAG with Azure AI Foundry 📖 Azure AI Search hybrid search

---

3. Multi-Agent Orchestration Architecture

Multiple specialized agents coordinated by a supervisor agent.

graph TB
    subgraph Client
        U[User]
    end

    subgraph Orchestration["Agent Orchestration Layer"]
        S[Supervisor Agent]
        S --> R1[Research Agent]
        S --> R2[Analysis Agent]
        S --> R3[Writing Agent]
        S --> R4[Code Agent]
    end

    subgraph Tools["Agent Tools"]
        R1 --> T1[Bing Search]
        R1 --> T2[File Search]
        R2 --> T3[Code Interpreter]
        R2 --> T4[Custom API]
        R3 --> T5[File Search]
        R4 --> T6[Code Interpreter]
        R4 --> T7[Custom Functions]
    end

    subgraph AI_Foundry["Azure AI Foundry"]
        M1[GPT-4o - Supervisor]
        M2[GPT-4o mini - Research]
        M3[GPT-4o - Analysis]
        M4[GPT-4o - Writing]
    end

    U --> S
    S --> M1
    R1 --> M2
    R2 --> M3
    R3 --> M4
    R4 --> M3

Components

Component	Purpose	Model
Supervisor Agent	Routes tasks, aggregates results	GPT-4o (high reasoning)
Research Agent	Web search, document retrieval	GPT-4o mini (cost-efficient)
Analysis Agent	Data analysis, calculations	GPT-4o (with Code Interpreter)
Writing Agent	Content generation, formatting	GPT-4o
Code Agent	Code generation, debugging	GPT-4o (with Code Interpreter)

Orchestration Patterns

Pattern	Description	Use Case
Sequential	Agent A → Agent B → Agent C	Step-by-step workflows
Parallel	Agent A + B + C simultaneously	Independent sub-tasks
Supervisor	Router agent delegates to specialists	Complex, multi-skill tasks
Handoff	Agent A transfers to Agent B	Escalation, specialization

Implementation Notes

• Use the Azure AI Agent Service for agent lifecycle management.
• Assign different models to agents based on task complexity.
• Implement guardrails at the supervisor level to prevent loops.
• Use threads to maintain conversation state per agent.
• Set maximum iteration limits to prevent runaway agents.

📖 Azure AI Agent Service

---

4. Fine-Tuning Pipeline

End-to-end pipeline for fine-tuning, evaluating, and deploying custom models.

graph LR
    subgraph DataPrep["1. Data Preparation"]
        A1[Raw Data] --> A2[Data Cleaning]
        A2 --> A3[Format to JSONL]
        A3 --> A4[Train/Validation Split]
    end

    subgraph Training["2. Fine-Tuning"]
        A4 --> B1[Upload Training Data]
        B1 --> B2[Configure Hyperparameters]
        B2 --> B3[Submit Fine-Tuning Job]
        B3 --> B4[Monitor Training Metrics]
    end

    subgraph Evaluation["3. Evaluation"]
        B4 --> C1[Deploy Fine-Tuned Model]
        C1 --> C2[Run Evaluation Suite]
        C2 --> C3{Quality Gate}
        C3 -->|Pass| C4[Approve for Production]
        C3 -->|Fail| A1
    end

    subgraph Deployment["4. Production"]
        C4 --> D1[Production Deployment]
        D1 --> D2[A/B Testing]
        D2 --> D3[Monitor Performance]
        D3 -->|Drift Detected| A1
    end

Components

Stage	Component	Details
Data Prep	Python scripts	pandas, data validation, JSONL formatting
Training	AI Foundry Fine-Tuning	GPT-4o mini, configurable epochs/LR/batch size
Evaluation	azure-ai-evaluation	Groundedness, Relevance, custom metrics
Deployment	AI Foundry Deployment	Standard PAYG or PTU
Monitoring	Azure Monitor	Token usage, latency, error rate

Training Data Format (JSONL)

{"messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}

Key Considerations

• Minimum 50 examples recommended; 500+ for production quality.
• Always hold out 20% for validation.
• Compare fine-tuned model against base model on the same evaluation set.
• Fine-tuning cost = training tokens × per-token training rate.
• Consider fine-tuning GPT-4o mini first (cheaper) before GPT-4o.

📖 Fine-tuning in Azure AI Foundry 📖 Prepare training data

---

5. MLOps CI/CD Pipeline for AI Foundry

Automated build, evaluate, and deploy pipeline using GitHub Actions.

graph LR
    subgraph Development
        A1[Developer] --> A2[Git Push]
    end

    subgraph CI["CI Pipeline (GitHub Actions)"]
        A2 --> B1[Lint & Unit Test]
        B1 --> B2[Build Application]
        B2 --> B3[Deploy to Staging]
    end

    subgraph Evaluation["Automated Evaluation"]
        B3 --> C1[Run Eval Dataset]
        C1 --> C2[Compute Metrics]
        C2 --> C3{Quality Gate<br/>Groundedness > 4.0<br/>Relevance > 4.0}
    end

    subgraph CD["CD Pipeline"]
        C3 -->|Pass| D1[Deploy to Production]
        D1 --> D2[Blue/Green Switch]
        D2 --> D3[Smoke Tests]
        D3 --> D4[Monitor]
    end

    subgraph Rollback
        C3 -->|Fail| E1[Block Deployment]
        D3 -->|Fail| E2[Rollback to Previous]
    end

Pipeline Stages

Stage	Tools	Duration
Lint & Test	pytest, ruff, mypy	~2 min
Build	Docker, pip	~5 min
Deploy to Staging	azd deploy, az CLI	~5 min
Evaluation	azure-ai-evaluation SDK	~10 min
Quality Gate	Custom threshold checks	~1 min
Production Deploy	azd deploy, blue/green	~5 min
Smoke Tests	curl, pytest	~2 min

GitHub Actions Workflow (Simplified)

name: AI Foundry CI/CD
on:
  push:
    branches: [main]

jobs:
  build-and-evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: azure/login@v2
        with:
          creds: $
      - run: pip install -r requirements.txt
      - run: pytest tests/
      - run: python scripts/evaluate.py --threshold 4.0
      - run: azd deploy --environment production

📖 Evaluate and deploy with CI/CD

---

6. Content Safety Integration Pattern

Layered content safety with pre-processing, model-level, and post-processing filters.

graph TB
    subgraph Input["Input Layer"]
        A1[User Input] --> A2[Prompt Shield]
        A2 -->|Jailbreak Detected| A3[Block & Log]
        A2 -->|Safe| A4[Input Content Filter]
        A4 -->|Unsafe| A3
        A4 -->|Safe| A5[Input to Model]
    end

    subgraph Model["Model Layer"]
        A5 --> B1[GPT-4o]
        B1 --> B2[Model Output]
    end

    subgraph Output["Output Layer"]
        B2 --> C1[Output Content Filter]
        C1 -->|Unsafe| C2[Fallback Response]
        C1 -->|Safe| C3[Groundedness Check]
        C3 -->|Ungrounded| C4[Flag for Review]
        C3 -->|Grounded| C5[Return to User]
    end

    subgraph Monitoring["Monitoring"]
        A3 --> D1[Azure Monitor]
        C2 --> D1
        C4 --> D1
        D1 --> D2[Alert Rules]
        D2 --> D3[Security Team]
    end

Safety Layers

Layer	Component	What It Catches
Pre-Model	Prompt Shields	Jailbreak attempts, prompt injection
Pre-Model	Input Content Filter	Hate, violence, self-harm, sexual content in user input
Model-Level	System Prompt	Behavioral guardrails, scope limitation
Post-Model	Output Content Filter	Harmful content in model output
Post-Model	Groundedness Detection	Hallucinated or ungrounded claims
Monitoring	Azure Monitor + Alerts	Pattern detection, incident response

Content Filter Severity Levels

Level	Description	Default Action
Safe	No harmful content	Allow
Low	Minor concerns	Allow (configurable)
Medium	Moderate concerns	Block (configurable)
High	Severe concerns	Always block

📖 Content filtering in Azure AI Foundry 📖 Prompt Shields

---

7. Hub-Spoke Enterprise Deployment

Multi-team enterprise architecture with centralized governance.

graph TB
    subgraph CentralIT["Central IT (Hub)"]
        H[AI Foundry Hub]
        H --> KV[Key Vault]
        H --> SA[Storage Account]
        H --> AI[AI Services Connection]
        H --> NW[Virtual Network]
        H --> MON[Azure Monitor]
        H --> POL[Azure Policy]
    end

    subgraph TeamA["Team A (Project)"]
        PA[Project: Customer Service]
        PA --> DA1[GPT-4o Deployment]
        PA --> DA2[RAG Pipeline]
        PA --> DA3[Content Safety - Strict]
    end

    subgraph TeamB["Team B (Project)"]
        PB[Project: Internal Tools]
        PB --> DB1[GPT-4o mini Deployment]
        PB --> DB2[Agent Service]
        PB --> DB3[Content Safety - Standard]
    end

    subgraph TeamC["Team C (Project)"]
        PC[Project: Research]
        PC --> DC1[Llama 3 Deployment]
        PC --> DC2[Fine-Tuning Jobs]
        PC --> DC3[Evaluation Pipelines]
    end

    H --> PA
    H --> PB
    H --> PC

    subgraph Network["Network Security"]
        PE1[Private Endpoints]
        FW[Azure Firewall]
        NSG[Network Security Groups]
    end

    NW --> PE1
    NW --> FW
    NW --> NSG

Architecture Principles

Principle	Implementation
Centralized governance	Hub manages shared resources, policies, and network
Team autonomy	Each project has independent deployments and data
Shared resources	Key Vault, Storage, AI Services connection shared via Hub
Network isolation	Private endpoints, no public internet access
Cost attribution	Per-project cost tracking via resource tags
RBAC separation	Team members only access their project

RBAC Structure

Role	Scope	Assigned To
Owner	Hub	Platform team
Azure AI Developer	Project A	Team A members
Azure AI Developer	Project B	Team B members
Azure AI Developer	Project C	Team C members
Reader	Hub	All developers (visibility)

📖 AI Foundry Hub architecture 📖 Configure network security

---

8. Real-Time Streaming Architecture

Low-latency streaming architecture for real-time AI-powered applications.

graph LR
    subgraph Clients["Clients"]
        C1[Web App]
        C2[Mobile App]
        C3[API Consumer]
    end

    subgraph Gateway["API Layer"]
        AG[Azure API Management]
        AG --> RL[Rate Limiting]
        AG --> AUTH[OAuth 2.0 / Entra ID]
        AG --> CACHE[Response Cache]
    end

    subgraph Compute["Application Layer"]
        ACA[Azure Container Apps]
        ACA --> WS[WebSocket Handler]
        ACA --> SSE[SSE Handler]
        ACA --> Q[Azure Service Bus]
    end

    subgraph AI["AI Layer"]
        subgraph Foundry["Azure AI Foundry"]
            GPT[GPT-4o Streaming]
            EMB[Embedding Model]
            CF[Content Filter]
        end
        SEARCH[Azure AI Search]
    end

    subgraph State["State Management"]
        REDIS[Azure Cache for Redis]
        COSMOS[Azure Cosmos DB]
    end

    C1 -->|WebSocket| AG
    C2 -->|SSE| AG
    C3 -->|REST| AG
    AG --> ACA
    WS --> GPT
    SSE --> GPT
    ACA --> SEARCH
    ACA --> EMB
    GPT --> CF
    ACA --> REDIS
    ACA --> COSMOS
    Q --> ACA

Components

Component	Purpose	Why
API Management	Gateway, auth, rate limiting	Centralized traffic management
Container Apps	Application hosting	Auto-scale, WebSocket support
GPT-4o (Streaming)	Real-time text generation	stream=True for SSE output
Redis Cache	Session state, prompt cache	Sub-ms latency for repeated queries
Cosmos DB	Conversation history	Global distribution, low latency
Service Bus	Async task queue	Decouple long-running operations
AI Search	Knowledge retrieval	RAG for grounded responses

Streaming Implementation

# Server-Sent Events (SSE) with streaming
async def stream_response(request):
    async def event_generator():
        response = client.complete(
            model="gpt-4o",
            messages=messages,
            stream=True
        )
        for chunk in response:
            content = chunk.choices[0].delta.content
            if content:
                yield f"data: {json.dumps({'content': content})}\n\n"
        yield "data: [DONE]\n\n"

    return StreamingResponse(event_generator(), media_type="text/event-stream")

Performance Targets

Metric	Target
Time to first token	< 500ms
Token generation rate	50–100 tokens/sec
WebSocket connection time	< 200ms
Cache hit rate	> 60% for repeated queries
End-to-end P99 latency	< 3 seconds

📖 Streaming with Azure OpenAI 📖 Azure Container Apps with WebSockets

---

Architecture Selection Guide

Scenario	Recommended Architecture	Key Components
Simple chatbot / PoC	1. Basic Chat	App Service + GPT-4o
Knowledge Q&A over documents	2. Enterprise RAG	AI Search + GPT-4o + Blob Storage
Complex multi-step workflows	3. Multi-Agent	Agent Service + multiple models
Domain-specific language/style	4. Fine-Tuning Pipeline	Fine-tuning + Evaluation
Continuous delivery of AI apps	5. MLOps CI/CD	GitHub Actions + Evaluation SDK
Regulated/sensitive content	6. Content Safety	Layered filters + monitoring
Multi-team enterprise	7. Hub-Spoke	Hub + Projects + RBAC + VNet
Low-latency real-time apps	8. Real-Time Streaming	Container Apps + WebSocket + Redis

---

Further Reading

• Azure AI Foundry documentation
• Azure Architecture Center — AI
• Well-Architected Framework for AI
• Azure OpenAI Service best practices