Evaluating Generative AI Models Using Microsoft Foundry’s Continuous Evaluation Framework

In this article, we’ll explore how to design, configure, and operationalize model evaluation using Microsoft Foundry’s built-in capabilities and best practices. Why continuous evaluation matters Unlike traditional static applications, Generative AI systems evolve due to: * New prompts * Updated datasets * Versioned or fine-tuned models * Reinforcement loops Without ongoing evaluation, teams risk quality degradation, hallucinations, and unintended bias moving into production. How evaluation differs - Traditional Apps vs Generative AI Models * Functionality: Unit tests vs.

STEP 1 — Set up your evaluation project in microsoft foundry 1. Open Microsoft Foundry Portal → navigate to your workspace. Click “Evaluation” from the left navigation pane.

Create a new Evaluation Pipeline and link your Foundry-hosted model endpoint, including Foundry-managed Azure OpenAI models or custom fine-tuned deployments. Choose or upload your test dataset — e.g., sample prompts and expected outputs (ground truth). Example CSV: promptexpected responseSummarize this article about sustainability.A concise, factual summary without personal opinions.Generate a polite support response for a delayed shipment.Apologetic, empathetic tone acknowledging the delay.

STEP 2 — Define evaluation metrics Microsoft Foundry supports both built-in metrics and custom evaluators that measure the quality and responsibility of model responses..