Azure-AI_Foundry

Lesson 3: Prompt Flow

Overview

Prompt Flow is a development framework in Azure AI Foundry for building, testing, and deploying LLM-powered workflows. It provides a visual graph-based interface for connecting prompts, tools, and code into production-ready pipelines.

Official Documentation: Prompt Flow in Azure AI Foundry

---

What Is Prompt Flow?

Prompt Flow lets you build Directed Acyclic Graphs (DAGs) of steps — each step is a node that can be a prompt template, a Python function, or a tool call. The output of one node feeds into the next.

┌───────────┐    ┌──────────────┐    ┌───────────┐    ┌────────────┐
│  Input     │───▶│  Classify    │───▶│  Route    │───▶│  Generate  │
│  (user     │    │  (LLM node)  │    │  (Python  │    │  Response  │
│   query)   │    │              │    │   node)   │    │  (LLM node)│
└───────────┘    └──────────────┘    └───────────┘    └────────────┘

Why Use Prompt Flow?

Benefit	Description
Visual authoring	Build and debug flows with a graph UI
Modularity	Break complex logic into testable nodes
Evaluation	Built-in eval flows for quality, groundedness, relevance
Versioning	Track flow versions alongside code
Deployment	Deploy flows as managed endpoints with one click
Tracing	Built-in tracing for debugging and monitoring

---

Flow Types

1. Standard Flow

General-purpose flow for any LLM task. Use for:

• Data processing pipelines
• Content generation workflows
• Classification and routing logic
• Multi-step reasoning chains

2. Chat Flow

Optimized for conversational applications. Includes:

• Built-in chat_history and chat_input inputs
• Conversation context management
• Designed for multi-turn dialog

3. Evaluation Flow

Special flow type for measuring the quality of other flows. Use for:

• Scoring outputs (relevance, coherence, groundedness, fluency)
• Comparing model versions or prompt variations
• Running batch evaluations against test datasets

---

Creating Your First Flow

Option 1: In the Azure AI Foundry Portal

1. Navigate to your AI Foundry project
2. Click Prompt flow in the left navigation
3. Click + Create → choose a flow type (Standard, Chat, or Evaluation)
4. Use one of the built-in templates or start from scratch
5. Add and connect nodes in the visual editor

Option 2: Using the Prompt Flow SDK

# Install the prompt flow SDK
# pip install promptflow promptflow-tools

from promptflow.core import tool

@tool
def classify_intent(query: str) -> str:
    """Classify user intent from a query string."""
    query_lower = query.lower()
    if any(word in query_lower for word in ["price", "cost", "discount", "buy"]):
        return "sales"
    elif any(word in query_lower for word in ["broken", "error", "fix", "help"]):
        return "support"
    elif any(word in query_lower for word in ["return", "refund", "exchange"]):
        return "returns"
    else:
        return "general"

Option 3: Using YAML Flow Definition

Create a flow.dag.yaml file:

$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json

inputs:
  user_query:
    type: string
    default: "How much does the Pro plan cost?"

outputs:
  response:
    type: string
    reference: ${generate_response.output}

nodes:
  - name: classify_intent
    type: python
    source:
      type: code
      path: classify_intent.py
    inputs:
      query: ${inputs.user_query}

  - name: generate_response
    type: llm
    source:
      type: code
      path: generate_response.jinja2
    inputs:
      deployment_name: gpt-4o-prod
      intent: ${classify_intent.output}
      query: ${inputs.user_query}
    connection: my_aoai_connection
    api: chat

---

Node Types

LLM Node

Calls a language model with a prompt template (Jinja2 format):

generate_response.jinja2:

system:
You are a helpful assistant for Contoso Electronics.
The user's intent has been classified as: 

Respond appropriately based on the intent:
- sales: Provide pricing information and upsell relevant products.
- support: Ask clarifying questions and provide troubleshooting steps.
- returns: Explain the return policy and guide the process.
- general: Answer the question helpfully.

user:

Python Node

Custom logic using Python functions:

from promptflow.core import tool

@tool
def route_by_intent(intent: str, query: str) -> dict:
    """Route the query based on classified intent."""
    routing_config = {
        "sales": {
            "temperature": 0.7,
            "system_context": "Focus on product benefits and pricing.",
            "max_tokens": 500
        },
        "support": {
            "temperature": 0.3,
            "system_context": "Be precise and technical. Ask diagnostic questions.",
            "max_tokens": 800
        },
        "returns": {
            "temperature": 0.2,
            "system_context": "Follow the return policy exactly. Be empathetic.",
            "max_tokens": 400
        },
        "general": {
            "temperature": 0.5,
            "system_context": "Be helpful and concise.",
            "max_tokens": 300
        }
    }
    config = routing_config.get(intent, routing_config["general"])
    config["query"] = query
    config["intent"] = intent
    return config

Prompt Node

A simplified node that wraps a prompt template without full LLM configuration — useful for building prompt strings that feed into other nodes.

---

Testing Flows

Interactive Testing in Portal

1. Click the Run button in the flow editor
2. Provide test inputs in the input panel
3. View outputs at each node — click any node to see its input/output
4. Use the trace view to debug timing and data flow

Batch Testing with a Dataset

from promptflow.client import PFClient

pf = PFClient()

# Run flow against a test dataset
run = pf.run(
    flow="./my_flow",
    data="./test_data.jsonl",  # JSONL file with test inputs
    column_mapping={
        "user_query": "${data.question}"
    },
    stream=True
)

# Get results
details = pf.get_details(run)
print(details.head(10))

test_data.jsonl:

{"question": "How much does the Pro plan cost?"}
{"question": "My laptop screen is flickering."}
{"question": "I want to return my headphones."}
{"question": "What products do you sell?"}
{"question": "Can I get a discount for buying in bulk?"}

Unit Testing Individual Nodes

import pytest
from classify_intent import classify_intent

def test_classify_sales():
    assert classify_intent("How much does it cost?") == "sales"

def test_classify_support():
    assert classify_intent("My device is broken") == "support"

def test_classify_returns():
    assert classify_intent("I want to return this") == "returns"

def test_classify_general():
    assert classify_intent("What are your business hours?") == "general"

---

Evaluation Flows

Built-in Evaluation Metrics

Azure AI Foundry provides built-in evaluation flows for common quality metrics:

Metric	What It Measures
Groundedness	Is the response supported by the provided context?
Relevance	Does the response address the user’s question?
Coherence	Is the response logically consistent and well-structured?
Fluency	Is the response grammatically correct and natural?
Similarity	How similar is the response to a ground truth answer?
F1 Score	Token-level overlap with ground truth

Running an Evaluation

from promptflow.client import PFClient

pf = PFClient()

# Run the main flow first
main_run = pf.run(
    flow="./my_chat_flow",
    data="./eval_dataset.jsonl",
    column_mapping={"user_query": "${data.question}"}
)

# Run evaluation against the main flow's outputs
eval_run = pf.run(
    flow="./evaluation_flow",
    data="./eval_dataset.jsonl",
    run=main_run,  # Evaluate against this run's outputs
    column_mapping={
        "question": "${data.question}",
        "answer": "${run.outputs.response}",
        "context": "${data.context}",
        "ground_truth": "${data.expected_answer}"
    }
)

# View evaluation results
metrics = pf.get_metrics(eval_run)
print(f"Groundedness: {metrics['groundedness']:.2f}")
print(f"Relevance:    {metrics['relevance']:.2f}")
print(f"Coherence:    {metrics['coherence']:.2f}")

---

Deploying Flows

Deploy as a Managed Endpoint

1. In the flow editor, click Deploy
2. Choose deployment settings:
   • Endpoint name — unique identifier
   • VM size — compute for inference
   • Instance count — number of replicas
   • Environment — Python dependencies
3. Click Deploy

Deploy with SDK

from azure.ai.ml import MLClient
from azure.ai.ml.entities import (
    ManagedOnlineEndpoint,
    ManagedOnlineDeployment,
    Model,
    Environment,
)
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()
ml_client = MLClient(credential, "your-sub-id", "your-rg", "your-workspace")

# Create endpoint
endpoint = ManagedOnlineEndpoint(
    name="my-flow-endpoint",
    description="Customer support chat flow",
    auth_mode="key"
)
ml_client.online_endpoints.begin_create_or_update(endpoint).result()

# Create deployment
deployment = ManagedOnlineDeployment(
    name="v1",
    endpoint_name="my-flow-endpoint",
    model=Model(path="./my_flow"),
    environment=Environment(
        image="mcr.microsoft.com/azureml/promptflow/promptflow-runtime:latest",
        conda_file="./my_flow/requirements.txt"
    ),
    instance_type="Standard_DS3_v2",
    instance_count=1
)
ml_client.online_deployments.begin_create_or_update(deployment).result()

Calling a Deployed Flow

import requests

endpoint_url = "https://my-flow-endpoint.eastus2.inference.ml.azure.com/score"
api_key = "your-endpoint-key"

response = requests.post(
    endpoint_url,
    headers={
        "Content-Type": "application/json",
        "Authorization": f"Bearer {api_key}"
    },
    json={"user_query": "How much does the Pro plan cost?"}
)

print(response.json())

---

Flow Development Best Practices

Project Structure

my_flow/
├── flow.dag.yaml           # Flow definition (nodes, connections)
├── classify_intent.py      # Python node
├── route_by_intent.py      # Python node
├── generate_response.jinja2 # LLM prompt template
├── requirements.txt        # Python dependencies
├── .env                    # Connection strings (gitignored)
├── data/
│   ├── test_data.jsonl     # Test inputs
│   └── eval_data.jsonl     # Evaluation dataset
└── tests/
    └── test_nodes.py       # Unit tests for Python nodes

Connection Management

# Connections store credentials for LLM endpoints, search services, etc.
# Create connections in the portal or via CLI:
# az ml connection create --file connection.yaml --resource-group your-rg --workspace-name your-ws

# connection.yaml example:
name: my_aoai_connection
type: azure_open_ai
api_base: https://your-resource.openai.azure.com/
api_type: azure
api_version: 2024-12-01-preview

---

Tips

💡 Start simple. Build a 2-node flow (input → LLM → output) first. Add complexity (routing, evaluation, error handling) only after the basic path works.

💡 Test each node independently before connecting them. A failing flow is harder to debug than a failing function.

💡 Use evaluation flows in CI/CD. Run evaluations automatically on PR merges to catch quality regressions before they hit production.

💡 Version your flows in Git. flow.dag.yaml and all node files are plain text — treat them like application code.

💡 Use the tracing feature to understand latency bottlenecks. The visual trace view shows exactly how long each node takes and what data flows between them.

💡 Prompt Flow vs direct SDK calls: Use Prompt Flow when you need multi-step orchestration, built-in evaluation, or visual debugging. For simple single-call scenarios, direct SDK calls are simpler and faster.

---

Summary

• Prompt Flow provides a graph-based framework for building LLM pipelines
• Three flow types: Standard (general), Chat (conversational), Evaluation (quality measurement)
• Nodes can be LLM calls, Python functions, or tool integrations
• Built-in evaluation metrics measure groundedness, relevance, coherence, and fluency
• Flows can be deployed as managed endpoints for production use
• Treat flows as code — version control, test, and evaluate them in CI/CD

---

Next: Knowledge Check