Azure-AI_Foundry

Lesson 1: Endpoint Management

Real-Time vs Batch Endpoints

Feature Real-Time Endpoints Batch Endpoints
Latency Milliseconds Minutes to hours
Use case Chat, interactive apps Document processing, bulk scoring
Scaling Auto-scale on load Job-based compute
Cost Pay for uptime Pay per job

Deploying a Real-Time Endpoint

from azure.ai.ml import MLClient
from azure.ai.ml.entities import ManagedOnlineEndpoint, ManagedOnlineDeployment
from azure.identity import DefaultAzureCredential

ml_client = MLClient(
    DefaultAzureCredential(),
    subscription_id="<sub-id>",
    resource_group_name="<rg>",
    workspace_name="<ai-project>",
)

# Create endpoint with Azure AD auth (recommended over key-based)
endpoint = ManagedOnlineEndpoint(
    name="genai-endpoint",
    auth_mode="aad_token",
    description="Production GenAI endpoint",
)
ml_client.online_endpoints.begin_create_or_update(endpoint).result()

# Create deployment
deployment = ManagedOnlineDeployment(
    name="blue",
    endpoint_name="genai-endpoint",
    model="azureml://registries/azureml/models/gpt-4o/versions/1",
    instance_type="Standard_DS3_v2",
    instance_count=2,
)
ml_client.online_deployments.begin_create_or_update(deployment).result()

# Route 100% traffic to this deployment
endpoint.traffic = {"blue": 100}
ml_client.online_endpoints.begin_create_or_update(endpoint).result()

Authentication: Key vs Azure AD

# Azure AD token auth (recommended for production)
from azure.identity import DefaultAzureCredential
import requests

credential = DefaultAzureCredential()
token = credential.get_token("https://ml.azure.com/.default")
headers = {
    "Authorization": f"Bearer {token.token}",
    "Content-Type": "application/json",
}
url = "https://genai-endpoint.eastus.inference.ml.azure.com/score"
response = requests.post(url, headers=headers, json={"prompt": "Hello"})

Tip: Always use Azure AD (Entra ID) authentication in production. Key-based auth is acceptable only for quick prototyping.

Auto-Scaling Configuration

deployment = ManagedOnlineDeployment(
    name="blue",
    endpoint_name="genai-endpoint",
    model="azureml://registries/azureml/models/gpt-4o/versions/1",
    instance_type="Standard_DS3_v2",
    instance_count=2,
    scale_settings={
        "type": "target_utilization",
        "min_instances": 2,
        "max_instances": 8,
        "target_utilization_percentage": 70,
        "polling_interval": 10,
    },
)

A/B Testing with Traffic Splitting

# Deploy a second version (green)
green_deployment = ManagedOnlineDeployment(
    name="green",
    endpoint_name="genai-endpoint",
    model="azureml://registries/azureml/models/gpt-4o/versions/2",
    instance_type="Standard_DS3_v2",
    instance_count=2,
)
ml_client.online_deployments.begin_create_or_update(green_deployment).result()

# Split traffic: 90% blue, 10% green
endpoint.traffic = {"blue": 90, "green": 10}
ml_client.online_endpoints.begin_create_or_update(endpoint).result()

# Monitor metrics, then gradually shift traffic
# 50/50 --> 0/100 --> delete old deployment

Tip: Use mirrored traffic (shadow mode) to test a new deployment with production data without affecting users.

References