35+ practical tips organized by category. Each tip includes a brief explanation and links to official Microsoft documentation where relevant.
Explore models, prompts, and data connections in the portal at ai.azure.com before investing in code. The Playground lets you validate your approach in minutes. Export working configurations to code when ready.
📖 AI Foundry portal quickstart
DefaultAzureCredential EverywhereNever hard-code API keys. DefaultAzureCredential from azure-identity works seamlessly across local dev (using Azure CLI login), managed identity in production, and CI/CD pipelines with service principals.
from azure.identity import DefaultAzureCredential
credential = DefaultAzureCredential()
Use the Hub/Project hierarchy to isolate environments. A single Hub can host multiple Projects — one per environment or team. This gives you independent model deployments, data connections, and access control per environment.
📖 AI Foundry Hub and Project architecture
Always specify an explicit api-version in REST calls. The default version can change, potentially breaking your application. Use the latest GA version for stability.
api-version=2024-10-21 # Pinned, stable
GPT-4o mini is ~15× cheaper than GPT-4o. For classification, extraction, simple Q&A, and formatting tasks, GPT-4o mini often performs equally well at a fraction of the cost.
| Model | Input/1M tokens | Output/1M tokens |
|---|---|---|
| GPT-4o | $2.50 | $10.00 |
| GPT-4o mini | $0.15 | $0.60 |
If you’re on PTU, monitor your actual utilization. Over-provisioned PTU is wasted money. Under-provisioned PTU causes 429 errors. Start with PAYG, measure your steady-state load, then switch to PTU when patterns are predictable.
Always set max_tokens to a reasonable value. Without it, the model generates until it hits the context window, consuming unnecessary tokens.
response = client.complete(
messages=messages,
max_tokens=500 # Cap output to control cost
)
Managed Compute deployments charge per hour whether you’re using them or not. Serverless (PAYG) deployments have no idle cost but still consume quota. Delete deployments you’re not actively using.
az ai model deployment list --project <project> --hub <hub>
az ai model deployment delete --name <deployment> --project <project> --hub <hub>
Before committing to a deployment, estimate costs with the Azure Pricing Calculator. Account for both token usage and supporting services (AI Search, Storage, App Service).
If your application frequently asks the same questions (e.g., FAQ bot), implement a semantic cache layer with Azure Cache for Redis or a simple hash-based cache. This can reduce token costs by 40–60%.
Streaming reduces perceived latency dramatically. The user sees the first token in ~200ms instead of waiting 2–5 seconds for the full response.
for chunk in client.complete(messages=messages, stream=True):
print(chunk.choices[0].delta.content or "", end="", flush=True)
When embedding many documents, batch them into a single API call (up to 2,048 inputs). This is dramatically faster than sending one-at-a-time.
response = embedding_client.embed(
input=["doc1 text", "doc2 text", "doc3 text", ...], # Up to 2048
model="text-embedding-3-small"
)
Don’t default to the most powerful model. Match model capability to task complexity:
| Task | Recommended Model |
|---|---|
| Simple classification | GPT-4o mini |
| Complex reasoning | GPT-4o or o1 |
| Code generation | GPT-4o |
| Embeddings (cost) | text-embedding-3-small |
| Embeddings (quality) | text-embedding-3-large |
| Quick summaries | GPT-4o mini |
Every token costs money and adds latency. Trim conversation history, compress system prompts, and remove irrelevant context. Use a sliding window for multi-turn conversations instead of sending the entire history.
Deploy models in the same region as your application to minimize network latency. If your app is in East US, deploy models in East US — not Sweden Central.
Managed identity eliminates the risk of key leakage. Enable system-assigned managed identity on your App Service / Container App and grant it the Cognitive Services OpenAI User role.
# Enable managed identity
az webapp identity assign --name <app> --resource-group <rg>
# Grant role
az role assignment create \
--assignee <identity-principal-id> \
--role "Cognitive Services OpenAI User" \
--scope <ai-services-resource-id>
📖 Managed identities for Azure AI
In production, disable public network access on your AI Foundry Hub and use private endpoints. This ensures all traffic stays within your Azure virtual network.
📖 Configure private link for AI Foundry
Don’t give everyone Contributor on the Hub. Use the dedicated AI Foundry roles:
If you must use API keys (e.g., for third-party integrations), rotate them every 90 days. Store keys in Azure Key Vault and reference them via Key Vault references in App Service settings.
Enable diagnostic logging on your AI Services resource to track who is calling your models, what they’re sending, and how many tokens they’re consuming.
A good system prompt is the highest-leverage improvement you can make. Be specific about role, constraints, output format, and what not to do.
You are a customer service assistant for Contoso Electronics.
- Only answer questions about Contoso products.
- If you don't know, say "I don't have that information."
- Always include a product link when referencing a product.
- Never discuss competitor products.
- Respond in the same language as the user.
📖 System message best practices
Include 2–3 input/output examples in the prompt to demonstrate the expected format. This is often more effective than lengthy instructions.
User: What's the return policy for laptops?
Assistant: Our laptop return policy allows returns within 30 days of purchase with original packaging. [Source: Return Policy, Section 3.2]
User: Can I get a refund for a damaged screen?
Assistant: ...
For multi-step problems, ask the model to think step by step. This improves accuracy on math, logic, and multi-hop reasoning tasks.
Think through this step by step before giving your final answer.
When you need structured data, use JSON mode to guarantee valid JSON output. This eliminates parsing errors.
response = client.complete(
messages=messages,
response_format={"type": "json_object"},
)
data = json.loads(response.choices[0].message.content)
Vector search alone misses exact-match keywords. Keyword search alone misses semantic meaning. Combine both with semantic ranking for the best retrieval quality.
results = search_client.search(
search_text="user query", # Keyword
vector_queries=[vector_query], # Vector
query_type="semantic", # Semantic ranking
semantic_configuration_name="default"
)
There’s no one-size-fits-all chunk size. Start with 512 tokens, measure retrieval quality, then adjust:
Enrich chunks with metadata (document title, section header, page number, date) and use it for filtering. This improves retrieval precision dramatically.
{
"content": "The return policy allows...",
"title": "Customer Service Guide",
"section": "Returns",
"page": 12,
"last_updated": "2024-06-01"
}
After retrieving top-K results (e.g., 20), rerank them to select the most relevant subset (e.g., top 5). Azure AI Search semantic ranking does this automatically, or use a cross-encoder model for custom reranking.
The model uses tool descriptions to decide when to call them. Vague descriptions lead to incorrect tool selection.
# ❌ Bad
{"name": "search", "description": "Searches for things"}
# ✅ Good
{"name": "search_products", "description": "Search the Contoso product catalog by name, category, or price range. Returns product name, price, and availability."}
When a tool call fails, return a helpful error message to the agent instead of crashing. The agent can then retry or take an alternative action.
def get_weather(city: str) -> str:
try:
result = weather_api.get(city)
return json.dumps(result)
except Exception as e:
return json.dumps({"error": f"Could not fetch weather for {city}: {str(e)}"})
Agents can get stuck in loops. Always set a maximum number of tool-calling iterations (e.g., 10) and a timeout. If the agent exceeds the limit, return a graceful fallback response.
Restrict what agents can do via system prompts and tool availability. Don’t give an agent access to tools it doesn’t need. A customer service agent doesn’t need a code interpreter.
📖 Azure AI Agent Service overview
Don’t wait until production to evaluate. Create a golden dataset of 50–100 representative queries with expected answers early in development. Update it as you discover edge cases.
No single metric tells the whole story. Run Groundedness + Relevance + Coherence + Fluency together. Custom evaluators add domain-specific quality checks.
eval_result = evaluate(
data="test_data.jsonl",
evaluators={
"groundedness": GroundednessEvaluator(config),
"relevance": RelevanceEvaluator(config),
"coherence": CoherenceEvaluator(config),
}
)
Add evaluation as a quality gate in your deployment pipeline. If Groundedness or Relevance drops below a threshold (e.g., 4.0/5.0), fail the deployment. This prevents regressions from reaching production.
Azure OpenAI returns 429 (rate limit) errors under load. Always implement retries with exponential backoff. The Retry-After header tells you exactly how long to wait.
import time
from azure.core.exceptions import HttpResponseError
max_retries = 3
for attempt in range(max_retries):
try:
response = client.complete(messages=messages)
break
except HttpResponseError as e:
if e.status_code == 429:
wait = int(e.response.headers.get("Retry-After", 2 ** attempt))
time.sleep(wait)
else:
raise
Track token consumption in Azure Monitor. Set alerts for unexpected spikes — they could indicate a runaway prompt, a bug in your context window management, or abuse.
When updating a model version or switching from a base model to a fine-tuned model, deploy the new version alongside the old one. Route a percentage of traffic to the new version, monitor quality, then cut over fully.
Log inputs and outputs for debugging and quality analysis, but be careful with PII. Use Azure Monitor with data masking or a dedicated logging service. Never log sensitive data in plain text.
Each message in the history consumes tokens. For long conversations, implement a sliding window (last N messages) or summarize older messages. A 50-turn conversation can easily exceed the context window.
When a content filter blocks a request, the API returns a content_filter finish reason instead of an error code. Always check finish_reason — don’t assume every response contains usable content.
if response.choices[0].finish_reason == "content_filter":
# Handle blocked response
return "I'm unable to respond to that request."
Users will try to jailbreak your model. Test with prompt injection attacks, role-playing attempts, and boundary-pushing queries before going to production. Use Prompt Shields and content filters as defense.
AI Search (Basic: ~$70/mo), Managed Compute deployments, and fine-tuned models all incur charges when idle. Set up a cleanup checklist and use Azure Cost Management alerts.
If you indexed documents with text-embedding-3-large (3072 dimensions), you must query with the same model. Mixing embedding models produces garbage results. Be consistent.
Model versions get deprecated. Use configuration variables or environment variables for model names so you can update without code changes.
# ❌ Bad
model = "gpt-4o-2024-08-06"
# ✅ Good
model = os.environ.get("AZURE_OPENAI_MODEL", "gpt-4o")
| # | Tip | Category |
|---|---|---|
| 1 | Start in the Portal, then code | Getting Started |
| 2 | Use DefaultAzureCredential |
Getting Started |
| 3 | Separate projects per environment | Getting Started |
| 4 | Pin API versions | Getting Started |
| 5 | GPT-4o mini for simple tasks | Cost |
| 6 | Right-size PTU | Cost |
| 7 | Set max_tokens |
Cost |
| 8 | Delete unused deployments | Cost |
| 9 | Use pricing calculator | Cost |
| 10 | Cache repeated queries | Cost |
| 11 | Stream responses | Performance |
| 12 | Batch embeddings | Performance |
| 13 | Match model to task | Performance |
| 14 | Minimize input tokens | Performance |
| 15 | Use regional endpoints | Performance |
| 16 | Managed identity, not keys | Security |
| 17 | Private endpoints for prod | Security |
| 18 | Least-privilege RBAC | Security |
| 19 | Rotate API keys | Security |
| 20 | Audit with Azure Monitor | Security |
| 21 | Explicit system prompts | Prompt Engineering |
| 22 | Few-shot examples | Prompt Engineering |
| 23 | Chain-of-thought | Prompt Engineering |
| 24 | JSON mode for structured output | Prompt Engineering |
| 25 | Hybrid search for RAG | RAG |
| 26 | Experiment with chunk sizes | RAG |
| 27 | Add metadata to chunks | RAG |
| 28 | Reranking step | RAG |
| 29 | Clear tool descriptions | Agents |
| 30 | Graceful tool error handling | Agents |
| 31 | Max iterations for agents | Agents |
| 32 | Guardrails for scope | Agents |
| 33 | Build eval datasets early | Evaluation |
| 34 | Run multiple evaluators | Evaluation |
| 35 | Automate eval in CI/CD | Evaluation |
| 36 | Retry with exponential backoff | Deployment |
| 37 | Monitor token usage | Deployment |
| 38 | Blue/green deployments | Deployment |
| 39 | Log prompts carefully | Deployment |