Azure-AI_Foundry

Lesson 3: Versioning and Rollback

Model Versioning Strategies

from azure.ai.ml.entities import Model

# Register a new model version with traceability tags
model = Model(
    name="genai-chat-model",
    version="3",
    path="./model-artifacts/",
    description="Fine-tuned GPT-4o with improved groundedness",
    tags={"eval_score": "4.5", "dataset": "v2", "fine_tune_epochs": "3"},
)
ml_client.models.create_or_update(model)

# List all versions
for m in ml_client.models.list(name="genai-chat-model"):
    print(f"v{m.version}: {m.description}")

Prompt Versioning

Store prompts as versioned artifacts alongside your application code:

# prompt-config.yaml
prompts:
  system: "v2.0"
  rag_template: "v1.2"
  summarization: "v1.0"
model:
  name: "gpt-4o"
  version: "2024-08-06"
  temperature: 0.7
  max_tokens: 4096

Tip: Treat prompt versions like API versions. Breaking changes get a major version bump. Refinements get minor bumps.

Blue-Green Deployments

# Deploy new version as "green" with 0% traffic
green = ManagedOnlineDeployment(
    name="green",
    endpoint_name="genai-endpoint",
    model="azureml://models/genai-chat-model/versions/3",
    instance_type="Standard_DS3_v2",
    instance_count=2,
)
ml_client.online_deployments.begin_create_or_update(green).result()

# Smoke test green directly
ml_client.online_endpoints.invoke(
    endpoint_name="genai-endpoint",
    deployment_name="green",
    request_file="tests/smoke_test.json",
)

# Switch all traffic, then delete old deployment
endpoint.traffic = {"blue": 0, "green": 100}
ml_client.online_endpoints.begin_create_or_update(endpoint).result()

Canary with Gradual Traffic Shift

import time

traffic_schedule = [
    {"blue": 95, "green": 5},
    {"blue": 80, "green": 20},
    {"blue": 50, "green": 50},
    {"blue": 0,  "green": 100},
]

for stage in traffic_schedule:
    endpoint.traffic = stage
    ml_client.online_endpoints.begin_create_or_update(endpoint).result()
    print(f"Traffic: {stage}")
    time.sleep(900)  # 15 min per stage

    metrics = get_endpoint_metrics("genai-endpoint")
    if metrics["error_rate"] > 0.05 or metrics["p99_latency_ms"] > 5000:
        print("Metrics degraded - rolling back!")
        endpoint.traffic = {"blue": 100, "green": 0}
        ml_client.online_endpoints.begin_create_or_update(endpoint).result()
        break

Rollback Procedures

Immediate rollback (seconds):

az ml online-endpoint update --name genai-endpoint \
  --traffic "blue=100 green=0" \
  --resource-group rg-ai-prod --workspace-name ai-project

Model rollback (minutes):

az ml online-deployment update --name green \
  --endpoint-name genai-endpoint \
  --model azureml://models/genai-chat-model/versions/2 \
  --resource-group rg-ai-prod --workspace-name ai-project

Disaster Recovery Planning

Component Strategy RPO RTO
Model registry Geo-replicated 0 Minutes
Endpoints Multi-region active-passive Minutes < 30 min
Prompt configs Git (source of truth) 0 Minutes
Index data AI Search replicas Near 0 Minutes

Tip: Always maintain at least one previous working deployment. Automate rollback triggers based on error rate thresholds.

References