Azure-AI_Foundry

Lesson 2: CI/CD Pipelines for AI Foundry

CI/CD Architecture for Generative AI

Developer Push --> CI Pipeline --> Evaluation Gate --> CD Pipeline
                   Build & Test    Auto-evaluate       Deploy to
                   prompt configs  quality & safety     staging/prod

GitHub Actions Workflow

# .github/workflows/genai-cicd.yml
name: GenAI CI/CD Pipeline
on:
  push:
    branches: [main]
    paths: ['src/prompts/**', 'src/flows/**', 'infra/**']
  pull_request:
    branches: [main]

env:
  AZURE_RESOURCE_GROUP: rg-ai-prod
  AI_PROJECT_NAME: genai-project
  ENDPOINT_NAME: genai-endpoint

permissions:
  id-token: write
  contents: read

jobs:
  build-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install -r requirements.txt
      - run: pytest tests/ -v --tb=short
      - run: python scripts/validate_prompts.py

  evaluate:
    needs: build-and-test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: azure/login@v2
        with:
          client-id: $
          tenant-id: $
          subscription-id: $
      - name: Run AI evaluations
        run: |
          python scripts/run_evaluations.py \
            --project $ \
            --dataset tests/eval_dataset.jsonl \
            --threshold-groundedness 4.0 \
            --threshold-coherence 4.0
      - run: python scripts/check_eval_gate.py --results eval_results.json

  deploy-staging:
    needs: evaluate
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - uses: actions/checkout@v4
      - uses: azure/login@v2
        with:
          client-id: $
          tenant-id: $
          subscription-id: $
      - name: Deploy to staging
        run: |
          az ml online-deployment create \
            --file deploy/staging-deployment.yml \
            --resource-group $ \
            --workspace-name $-staging

  deploy-production:
    needs: deploy-staging
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: azure/login@v2
        with:
          client-id: $
          tenant-id: $
          subscription-id: $
      - name: Deploy canary (10% traffic)
        run: |
          az ml online-deployment create \
            --file deploy/prod-deployment.yml \
            --resource-group $ \
            --workspace-name $
          az ml online-endpoint update \
            --name $ \
            --traffic "blue=90 green=10" \
            --resource-group $ \
            --workspace-name $

Automated Evaluation Gate Script

# scripts/check_eval_gate.py
import json, sys

def check_gate(results_path: str) -> bool:
    with open(results_path) as f:
        results = json.load(f)
    thresholds = {"groundedness": 4.0, "coherence": 4.0, "relevance": 4.0, "safety_defect_rate": 0.02}
    passed = True
    for metric, threshold in thresholds.items():
        value = results.get(metric, 0)
        ok = value <= threshold if metric == "safety_defect_rate" else value >= threshold
        print(f"{'PASS' if ok else 'FAIL'} {metric}: {value:.2f} (threshold: {threshold})")
        if not ok:
            passed = False
    return passed

if __name__ == "__main__":
    if not check_gate(sys.argv[1]):
        sys.exit(1)

Environment Promotion Strategy

Environment Purpose Approval Traffic
dev Rapid iteration Auto Developer only
staging Integration testing Auto on eval pass QA team
production Live users Manual approval Canary then full

IaC with Bicep

param endpointName string
param location string = resourceGroup().location

resource endpoint 'Microsoft.MachineLearningServices/workspaces/onlineEndpoints@2024-04-01' = {
  name: '${projectName}/${endpointName}'
  location: location
  properties: { authMode: 'AADToken' }
}

Secrets Management

Tip: Never store API keys in code. Use GitHub OIDC with Azure Workload Identity Federation for keyless CI/CD authentication. Store runtime secrets in Azure Key Vault.

References