Building AI responsibly goes beyond content filtering — it requires organizational governance, regulatory compliance, and systematic processes for reviewing, deploying, and monitoring AI systems. This lesson covers the frameworks, tools, and best practices for governing AI at enterprise scale.
📖 Reference: Microsoft Responsible AI
A robust AI governance framework includes five pillars:
┌─────────────────────────────────────────────────────────┐
│ AI GOVERNANCE FRAMEWORK │
├─────────────┬─────────────┬─────────────┬──────────────┤
│ Policy & │ Risk │ Technical │ People & │
│ Standards │ Assessment │ Controls │ Process │
├─────────────┼─────────────┼─────────────┼──────────────┤
│ AI use │ Impact │ Content │ Review │
│ policies │ assessments │ filters │ boards │
│ Data │ Risk │ Logging & │ Training & │
│ governance │ registers │ auditing │ awareness │
│ Model │ Bias │ Access │ Incident │
│ standards │ testing │ controls │ response │
└─────────────┴─────────────┴─────────────┴──────────────┘
# Example: AI deployment request tracking
ai_deployment_request = {
"application_name": "Customer Service Bot",
"model": "gpt-4o",
"use_case": "Automated customer support responses",
"data_classification": "Confidential",
"risk_level": "Medium",
"content_filters": "Default Microsoft filters + custom blocklist",
"data_residency": "EU (West Europe)",
"review_status": "Pending",
"reviewer": "AI Ethics Board",
"requested_by": "Product Team Alpha",
"date_submitted": "2024-11-15",
"impact_assessment_url": "https://internal.company.com/ai-reviews/cs-bot-001"
}
# When creating Azure OpenAI resources, choose regions that match
# your data residency requirements
# Example: Ensure EU data stays in EU
resource_config = {
"resource_group": "rg-ai-eu",
"location": "westeurope", # Data processed in EU
"kind": "OpenAI",
"sku": "S0",
"properties": {
"networkAcls": {
"defaultAction": "Deny" # Restrict network access
}
}
}
# Azure AI Foundry project configuration
project_config = {
"hub_name": "ai-hub-eu",
"location": "westeurope",
"storage_account": "staiprojecteu", # EU storage
"key_vault": "kv-ai-project-eu", # EU key vault
"description": "EU-compliant AI project for GDPR-regulated workloads"
}
📖 Reference: Azure data residency
Microsoft publishes Transparency Notes for each AI service, documenting:
For fine-tuned or custom models, document:
model_card = {
"model_name": "customer-intent-classifier-v2",
"base_model": "gpt-4o-mini",
"fine_tuning_date": "2024-10-01",
"training_data": {
"source": "Internal customer support tickets (anonymized)",
"size": "50,000 labeled examples",
"date_range": "2023-01 to 2024-09",
"demographics": "Global customer base, English-only",
"known_gaps": "Limited representation of APAC region queries"
},
"evaluation_results": {
"accuracy": 0.94,
"f1_score": 0.92,
"bias_assessment": "Tested across 8 demographic groups, max variance 3%",
"failure_modes": [
"Struggles with multi-intent queries",
"Lower accuracy on technical jargon"
]
},
"intended_use": "Routing customer support tickets to appropriate teams",
"out_of_scope": "Not for making financial decisions or medical advice",
"risk_level": "Low — assists human agents, does not auto-respond",
"review_frequency": "Quarterly performance review, annual bias audit"
}
📖 Reference: Transparency Notes for Azure AI
import logging
import json
from datetime import datetime, timezone
class AIAuditLogger:
"""Structured audit logging for AI interactions."""
def __init__(self):
self.logger = logging.getLogger("ai_audit")
handler = logging.FileHandler("ai_audit.log")
handler.setFormatter(logging.Formatter('%(message)s'))
self.logger.addHandler(handler)
self.logger.setLevel(logging.INFO)
def log_interaction(self, request_id: str, user_id: str,
model: str, prompt: str, response: str,
content_filter_results: dict,
tokens_used: dict, latency_ms: float):
"""Log a complete AI interaction for auditing."""
audit_record = {
"timestamp": datetime.now(timezone.utc).isoformat(),
"request_id": request_id,
"user_id": user_id,
"model": model,
"prompt_length": len(prompt),
"response_length": len(response),
"content_filter": content_filter_results,
"tokens": tokens_used,
"latency_ms": latency_ms,
"prompt_hash": hash(prompt), # For deduplication, not the actual prompt
"safety_flags": self._check_safety_flags(content_filter_results)
}
self.logger.info(json.dumps(audit_record))
def _check_safety_flags(self, filter_results: dict) -> list:
flags = []
for category, result in filter_results.items():
if result.get("severity", 0) >= 2:
flags.append(f"{category}:severity-{result['severity']}")
return flags
def log_safety_event(self, request_id: str, event_type: str,
details: str):
"""Log safety-related events (blocks, jailbreak attempts, etc.)."""
event = {
"timestamp": datetime.now(timezone.utc).isoformat(),
"request_id": request_id,
"event_type": event_type,
"details": details,
"severity": "HIGH" if "jailbreak" in event_type else "MEDIUM"
}
self.logger.info(json.dumps(event))
| ✅ Log | ❌ Do NOT Log |
|---|---|
| Request/response metadata | Full user prompts (PII risk) |
| Content filter results | Personal identifiable information |
| Token usage and costs | Authentication credentials |
| Safety events and blocks | System prompt contents |
| Model version and deployment | Internal API keys |
| Latency and performance data | Raw user data without consent |
## Responsible AI Pre-Deployment Review
### 1. Use Case Assessment
- [ ] Use case clearly defined and documented
- [ ] Impact assessment completed
- [ ] Risk level classified (Low / Medium / High / Critical)
- [ ] Stakeholders identified and consulted
### 2. Technical Controls
- [ ] Content filters configured and tested
- [ ] Prompt shields enabled
- [ ] Custom blocklists for domain-specific terms
- [ ] Rate limiting configured
- [ ] Input/output logging enabled
### 3. Data Governance
- [ ] Data residency requirements met
- [ ] PII handling documented
- [ ] Data retention policy defined
- [ ] User consent mechanisms in place
### 4. Testing & Evaluation
- [ ] Red-team testing completed
- [ ] Bias evaluation across demographic groups
- [ ] Groundedness evaluation (for RAG applications)
- [ ] Performance benchmarks documented
### 5. Operational Readiness
- [ ] Monitoring dashboards configured
- [ ] Alert thresholds set for safety events
- [ ] Incident response plan documented
- [ ] Human escalation path defined
- [ ] Rollback procedure tested
| Risk Level | Requirements | AI Foundry Examples |
|---|---|---|
| Unacceptable | Banned (social scoring, manipulation) | N/A — do not build these |
| High Risk | Conformity assessment, registration | HR screening, credit scoring |
| Limited Risk | Transparency obligations | Chatbots, content generation |
| Minimal Risk | No specific requirements | Spam filters, recommendations |
📖 Reference: Azure compliance offerings
AIAuditLogger and generate sample audit recordsNext: Knowledge Check