AI
Why Azure AI Foundry Evaluations Needed a Governance Layer
Azure AI Foundry already runs quality and safety evaluations for generative AI applications. What it lacked was a governance layer: a way for compliance officers and developers to agree on which evaluations to run and where results feed into regulatory evidence. That gap is what Cydas and Credo AI fill. Both platforms integrate directly with Foundry, pulling model metadata on one side and pushing evaluation results back to risk registers on the other.
Linking an AI System to Its Inherited Model Risks
Mary Halte, CEO at Cydas, ran the demo as an AI product owner at a financial services company building an HR chatbot on GPT-4o Mini. After registering the system in the Cydas inventory, she linked it to the model in Azure AI Foundry’s model catalog. That link pulled OpenAI’s published risk list for GPT-4o Mini directly into her system’s risk register, pre-populated. The deployment, task type, and inherited risks were all ready before any evaluation ran.
Auto-Generated Evaluation Plans and Red-Team Results
With the system registered, Cydas generated 8 evaluation plans automatically. The plans covered question-answering quality and red-team scenarios. Mary activated a red-team plan for harmful content, and Cydas produced a Cydas notebook generation for harmful-content red team (06:47) with attack strategies, risk category filters, and Azure subscription details already filled in. After the team ran the notebook, results flowed back into Cydas. The report flagged hateful and unfair content as underperforming, and Mary showed a before/after comparison from a previous remediation run.
From Governance Questionnaire to Runnable Evaluator Code
Eric Aldana registered a retail chatbot in the Credo AI platform, selecting GPT-4o Mini from the Foundry model catalog and answering a short questionnaire. Two answers, content safety not tested and no factual-accuracy audit process, were enough to trigger the governance plan. The plan identified EU AI Act transparency requirements and flagged risks for false information and toxic content. Opening any requirement revealed a Credo AI Python evaluator snippet for groundedness (15:14). Developers paste it into their IDE, run it, and results sync back to Credo AI alongside an acceptable-range check against the governance plan.
Notable Quotes
AI applications faster with the highest degree of trust. Eric Aldana · ▶ 10:18
governance and development, they exist in two different worlds, right? Eric Aldana · ▶ 11:32
Separated by language, tools, and priorities. Eric Aldana · ▶ 11:35
Key Takeaways
- Cydas pulls GPT-4o Mini’s published risk list from the Azure AI Foundry model catalog directly into a product owner’s risk register.
- Cydas generates 8 context-specific evaluation plans and maps red-team results back to specific compliance risks.
- Credo AI turns a short questionnaire into a governance plan and Python code snippets that run Foundry evaluators from the IDE.