Deploying a model is just the beginning. Production AI workloads require ongoing management — monitoring performance, scaling capacity, planning for model retirements, managing quotas, and optimizing costs. This lesson covers the operational practices for keeping your deployments healthy and cost-effective.
| Metric | What It Tells You | Alert Threshold |
|---|---|---|
| Token Usage (TPM) | How much capacity you are consuming | >80% of allocated TPM |
| Request Count | Volume of API calls | Unusual spikes or drops |
| Latency (P50/P95/P99) | Response time distribution | P95 > 2s for chat models |
| HTTP 429 Rate | Rate-limit rejections | Any sustained 429s |
| HTTP 5xx Rate | Server errors | >0.1% error rate |
| Token Utilization (PTU) | Provisioned throughput usage | <50% (over-provisioned) or >90% |
| Content Filter Blocks | Safety filter rejections | Unexpected spikes |
Azure OpenAI Requests (by StatusCode, ModelDeploymentName)Azure OpenAI Token Transactions (by ModelDeploymentName)Azure OpenAI Processing Time (latency)from azure.monitor.query import MetricsQueryClient
from azure.identity import DefaultAzureCredential
from datetime import timedelta
credential = DefaultAzureCredential()
metrics_client = MetricsQueryClient(credential)
resource_id = (
"/subscriptions/your-sub-id"
"/resourceGroups/your-rg"
"/providers/Microsoft.CognitiveServices"
"/accounts/your-openai-resource"
)
# Query token usage over the last 24 hours
response = metrics_client.query_resource(
resource_uri=resource_id,
metric_names=["TokenTransaction"],
timespan=timedelta(days=1),
granularity=timedelta(hours=1),
)
for metric in response.metrics:
for ts in metric.timeseries:
for data_point in ts.data:
if data_point.total:
print(f"{data_point.timestamp}: {data_point.total:.0f} tokens")
# Using Azure CLI to create an alert rule for 429 errors
az monitor metrics alert create \
--name "openai-rate-limit-alert" \
--resource-group "your-rg" \
--scopes "/subscriptions/.../Microsoft.CognitiveServices/accounts/your-resource" \
--condition "total Azure OpenAI Requests > 10" \
--dimension "StatusCode=429" \
--window-size 5m \
--evaluation-frequency 1m \
--action-group "your-action-group-id" \
--description "Alert when rate limiting occurs"
💡 Tip: Create alerts for both 429 (rate limit) and 5xx (server error) responses. A sudden spike in 429s often means your workload has grown beyond your quota allocation.
For Standard and Global Standard deployments, scaling means adjusting your Tokens Per Minute allocation:
from azure.mgmt.cognitiveservices import CognitiveServicesManagementClient
from azure.identity import DefaultAzureCredential
credential = DefaultAzureCredential()
mgmt_client = CognitiveServicesManagementClient(credential, "your-sub-id")
# Get current deployment
deployment = mgmt_client.deployments.get(
resource_group_name="your-rg",
account_name="your-openai-resource",
deployment_name="gpt-4o-prod"
)
# Scale up TPM (e.g., from 30K to 60K)
deployment.properties.current_capacity = 60 # in thousands
deployment.sku = deployment.sku # preserve SKU
result = mgmt_client.deployments.begin_create_or_update(
resource_group_name="your-rg",
account_name="your-openai-resource",
deployment_name="gpt-4o-prod",
deployment=deployment
).result()
print(f"Scaled to {result.properties.current_capacity}K TPM")
For Provisioned deployments, scaling means adjusting the number of PTUs:
For high-availability workloads, deploy the same model across multiple regions:
DEPLOYMENTS = [
{
"endpoint": "https://openai-eastus2.openai.azure.com/",
"key": "key-eastus2",
"deployment": "gpt-4o-prod",
"priority": 1
},
{
"endpoint": "https://openai-westus3.openai.azure.com/",
"key": "key-westus3",
"deployment": "gpt-4o-prod",
"priority": 2 # fallback
},
]
def get_client_with_fallback(deployments):
"""Try primary, fall back to secondary on 429 or 5xx."""
from openai import AzureOpenAI, RateLimitError, APIStatusError
sorted_deps = sorted(deployments, key=lambda d: d["priority"])
for dep in sorted_deps:
try:
client = AzureOpenAI(
azure_endpoint=dep["endpoint"],
api_key=dep["key"],
api_version="2024-12-01-preview"
)
response = client.chat.completions.create(
model=dep["deployment"],
messages=[{"role": "user", "content": "test"}],
max_tokens=5
)
return client, dep["deployment"]
except (RateLimitError, APIStatusError):
continue
raise Exception("All deployments unavailable")
Azure OpenAI models follow a predictable lifecycle:
Preview → GA → Deprecated (announced) → Retired (removed)
│ │ │ │
│ │ └─ ~6 months notice └─ Model no longer callable
│ └─ Stable for production
└─ May change without notice
from datetime import datetime
DEPLOYMENT_INVENTORY = [
{
"name": "gpt-4o-prod",
"model": "gpt-4o",
"version": "2024-11-20",
"retirement_date": datetime(2025, 8, 1),
"upgrade_path": "gpt-4o version 2025-xx-xx",
"owner": "platform-team",
},
{
"name": "gpt-4-legacy",
"model": "gpt-4",
"version": "0613",
"retirement_date": datetime(2025, 6, 1),
"upgrade_path": "gpt-4o",
"owner": "backend-team",
},
]
def check_retirement_status(inventory, alert_days=90):
"""Flag deployments approaching retirement."""
today = datetime.now()
for dep in inventory:
days_until = (dep["retirement_date"] - today).days
status = "🟢 OK" if days_until > alert_days else (
"🟡 Plan migration" if days_until > 30 else "🔴 URGENT"
)
print(f"{status} {dep['name']}: {days_until} days until retirement")
if days_until <= alert_days:
print(f" → Upgrade to: {dep['upgrade_path']}")
print(f" → Owner: {dep['owner']}")
check_retirement_status(DEPLOYMENT_INVENTORY)
💡 Tip: Maintain a deployment inventory with retirement dates. Review it monthly and start migration testing at least 90 days before retirement.
Azure OpenAI quotas are organized as:
Subscription
└── Region (e.g., East US 2)
└── Model (e.g., gpt-4o)
└── Deployment Type (e.g., GlobalStandard)
└── Quota limit (e.g., 1M TPM total)
├── Deployment A: 300K TPM
├── Deployment B: 200K TPM
└── Available: 500K TPM
from azure.mgmt.cognitiveservices import CognitiveServicesManagementClient
from azure.identity import DefaultAzureCredential
credential = DefaultAzureCredential()
mgmt_client = CognitiveServicesManagementClient(credential, "your-sub-id")
# List models and quota in a region
usages = mgmt_client.usages.list(location="eastus2")
for usage in usages:
if "OpenAI" in (usage.name.value or ""):
pct = (usage.current_value / usage.limit * 100) if usage.limit else 0
print(f"{usage.name.localized_value}: "
f"{usage.current_value}/{usage.limit} ({pct:.0f}%)")
💡 Tip: Request quota increases well before you need them. Don’t wait until you hit limits in production.
| Scenario | Instead Of | Use | Savings |
|---|---|---|---|
| Simple classification | GPT-4o | GPT-4o-mini | ~90% cheaper |
| FAQ / known patterns | GPT-4o | Fine-tuned GPT-4o-mini | ~85% cheaper |
| Embeddings (1536-dim sufficient) | text-embedding-3-large | text-embedding-3-small | ~80% cheaper |
| Structured extraction | GPT-4o | Phi-4 (Serverless) | ~70% cheaper |
def estimate_cost(input_tokens, output_tokens, model="gpt-4o"):
"""Estimate cost for a given number of tokens."""
pricing = {
"gpt-4o": {"input": 2.50, "output": 10.00}, # per 1M tokens
"gpt-4o-mini": {"input": 0.15, "output": 0.60},
"gpt-4.1": {"input": 2.00, "output": 8.00},
}
rates = pricing.get(model, pricing["gpt-4o"])
cost = (input_tokens / 1_000_000 * rates["input"] +
output_tokens / 1_000_000 * rates["output"])
return round(cost, 4)
# Compare costs for a batch job
daily_requests = 50_000
avg_input = 500 # tokens per request
avg_output = 200 # tokens per request
for model in ["gpt-4o", "gpt-4o-mini", "gpt-4.1"]:
daily_cost = estimate_cost(
daily_requests * avg_input,
daily_requests * avg_output,
model
)
print(f"{model}: ${daily_cost:.2f}/day (${daily_cost * 30:.2f}/month)")
import hashlib
import json
# Simple response cache to avoid redundant API calls
response_cache = {}
def cached_completion(client, messages, model, **kwargs):
"""Cache completions to avoid paying for identical requests."""
cache_key = hashlib.sha256(
json.dumps({"model": model, "messages": messages}, sort_keys=True).encode()
).hexdigest()
if cache_key in response_cache:
print("Cache hit — saved tokens!")
return response_cache[cache_key]
response = client.chat.completions.create(
model=model, messages=messages, **kwargs
)
response_cache[cache_key] = response
return response
Azure OpenAI’s Batch API offers 50% cost savings for workloads that can tolerate 24-hour turnaround:
# Batch API workflow:
# 1. Prepare a JSONL file with requests
# 2. Upload to Azure OpenAI
# 3. Create a batch job
# 4. Poll for completion (up to 24 hours)
# 5. Download results
#
# Ideal for:
# - Data labeling / classification
# - Bulk content generation
# - Offline analysis
# - Evaluation runs
Set up these views in Azure Cost Management:
💡 Set budget alerts at 50%, 80%, and 100% of your monthly AI budget. Catch runaway costs early.
💡 Tag your deployments with cost center, team, and application name. This makes cost allocation and chargeback much easier.
💡 Review your deployment inventory monthly. Delete unused deployments — they may hold quota even when idle.
💡 Use the Batch API for any workload that does not need real-time responses. The 50% discount adds up quickly for high-volume processing.
💡 Plan for model retirement at deploy time. When you create a deployment, note the model version and its expected retirement date in your team’s runbook.
Next: Knowledge Check