AI powered automation & multi-agent orchestration in Foundry

Hi everyone, my name is Sean Henry. I work on Microsoft Foundry on our developer tools and services like Microsoft Agent Framework, Semantic Kernel, Autogen, and the Foundry Agent Service. And today I'd like to talk about some of the new features we've added to Microsoft Foundry and the Agent Framework to help you build AI powered automation and multi-agent orchestrations. But before I dive into some of the new features, I'd like to look back on the last year. We've been talking about AI agents for about a year now. And with Microsoft Foundry and Microsoft Agent Framework, we've enabled you to build, deploy, and operate your agents in scale. And so some of the things I'm going to talk about today is how we've unified to a single Microsoft Agent Framework, how we're building multi-agent workflows. that allow you to visualize your workflows and have your agents work together, how you can deploy and manage and evaluate everything in one place with Foundry, how you can customize your experience for your users with channels and UI, and how you can apply compliance and governance to your A applications. And we've been doing this and we've been learning from our customers over the last year. And over a million agents and over 25,000 organizations are now using Foundry and Autogen and Symantec Kernel to build out scenarios like customer support and sales assistance, internal productivity, knowledge assistance, document research. You know, the list goes on and on of the things you can build when you start building your agents and then stringing them together into these multi-agent workflows and orchestration patterns. But as we were kind of talking to folks over the year and building with them and building with our partners We had a number of developer challenges that came up, you know things that we heard a lot was where do I start? How do I start building these these agents? What is the best way to get started? What framework should I use? I'm a developer what you know, how do I build these agents? How do I connect them to my other services? How do I make them work together if I build these agents that are talking to tools and talking to NCP servers? How do I make these agents talk together? How do I know what my agent's doing once I deploy it? How do I know how it's performing? Is it meeting expectations? Is it falling off over time? With LLMs, things are not always perfect all the time. They can be random and stochastic, and so you want to be able to continuously measure your systems. Finally, once I build my system and I've got it working, how do I deploy it and manage it at scale? And how do I help my users interface and talk to my agents and agent systems? So, we'll go through all of these problems and we'll talk about how we've approached those with Microsoft Agent Framework and Microsoft Foundry. So first of all, if you're not familiar with Microsoft Agent Framework, but you may be familiar with some of our libraries like Autogen and Semantic Kernel, the Foundry SDK, the Microsoft 365 Agent SDK. We've taken Microsoft Agent Foundry and made it our new framework. You can think of it as a successor to Symantec Kernel and Autogen. You know, the same team that brought you both of those libraries has built Microsoft Agent Framework as our kind of one unified Microsoft Procode experience for building agents and agent systems at Microsoft. And it takes the best from Autogen and Symantec Kernel. It builds on top of Foundry SDK and integrates really well with Microsoft Foundry, as well as with the M365 ecosystem through the M365 agent SDK. And what does it do? Well, it's your connector to all your AI services. And that's one of the important things about Microsoft agent framework is it's not just for connecting to Foundry services. Obviously, it connects to thousands of AI models in Foundry and the Foundry IQ and knowledge services that are available. And all the services from Azure but also connects to the broader ecosystem of AI services you can talk to models directly from open AI you can talk to models like from Google Gemini or Anthropic or AWS Bedrock Deep Seek all we have connectors for all of these as well as local models so you can use things like foundry local olama to connect the models that are hosted and running on your machine We also connect to the other AI services that you may need like knowledge and memory services for your vector stores that you need for knowledge bases for your agents and for agent memory services that are becoming more and more important to give your agents long-term contacts like Memzero. You can also connect directly to other agents, either directly to the Foundry agent service, agents hosted in Foundry, but also to other hosting services like Copilot Studio or Bedrock agents or LangGraph. And all this can be done either through direct connections in Microsoft Agent Framework or through our long list of open standards we support from OpenAPI and MCP for tools connections and logic apps for connections into that ecosystem for Microsoft as well as A to A for agent to agent communications. I'll show you some examples of some new UI frameworks that we've enabled as well including integration with chat kit and AGUI so you can quickly add UI on top of your agents. And finally, because We want to be enterprise ready. At-Milk Relation Framework supports the full set of open telemetry, Gen AI specification, so that you can take all of your output from your agent systems and send them either directly to Foundry or to any other service that supports OTEL so you can evaluate them. and monitor your agents in production. So all this is available today. We announced Microsoft Agent Framework in preview and it's available today in .NET and Python. And this allows you to build a full spectrum of agent solutions all the way from the most basic agents where you're really just querying an LLM to advanced single agents that do RAG to large multi-agent systems that are interacting together. Now one of the first questions you may have is like Sean you know I've been using semantic kernel and autogen like how do I migrate to agent framework and you don't worry we've got your back we have a great set of tools built directly into Visual Studio code that can take your existing semantic kernel and autogen projects and upgrade them. I've been using myself a lot of our partners have used these I've just been I've just been overwhelmed with how well this works in conjunction with Github Copilot. Just really a fantastic way to upgrade and migrate your code from from semantic kernel and autogen into the latest versions of Microsoft Agent Framework. But what I'd like to do now is I'd like to show you a quick demo and a tour of how you can build a sophisticated multi-agent system with Agent Framework. So I'm gonna pop over to Visual Studio Code here. And what I will show you here is, I have a project that I've built for event planning. and it has a bunch of agents in it. You can see how easy it is to create an agent with Microsoft Agent Framework. You can see here, this is a function that's for creating the agent for me, but to create an agent itself, it's just five or six lines of code, I can name the agent, give it a description, give it a prompt, and then connect it to all my agent tools. And I'll talk about a few things here. So one is the prompt, our prompt is defined over here. We can define it in text. We can have templates and slot filling and all those things that you may want to do with a prompt. We can give it tools. Here's an MCP tool that I've given it, a nice tool for sequential thinking. that I can use in order to think through a problem. And these are defined simply in my MCP file, so I connect those easily into Agent Framework. I can give it a set of built-in tools. For example, here's a code interpreter tool that I can use so it can execute code while it's thinking through a process. And if I was going too fast, we have hundreds of samples on the Agent Framework repo that you can go and take a look and use. But what I want to do is I want to actually connect all these agents together. I want to build a bunch of agents and connect them together. So in order to do that I built a workflow within agent framework. And so here's my workflow. I use a workflow builder. You know I give it a name and a description. And then I take all the agents that I've created here. My coordinator agent, my venue agent, my budget agent. And I connect them all together into a graph. And one thing I can do here in agent framework is I can connect not just agents, but I've wrapped with these agents here in this case in an executor which means I can add my own code in there as well. I don't just need to have an agent. I can run any arbitrary code. I can even not run an agent if I just have, you know, a node in my workflow that I want to execute. But all this is a little hard to see in code, so let's pop over to our new. our new dev UI here which shows us what this multi-agent system actually looks like. So in agent framework if you just type dev UI it'll bring up and execute your agents in our dev environment where you can inspect your agents and see how they're running. And you can see here I have all my agents that I've created. I've got my coordinator, I've got my venue specialist, my budget analyst, my logistics manager. And you can see I've all connected them together to my event coordinator. That's what I was building in code earlier. And from here I can run it, I can give it an input. I can say plan a corporate holiday party for 50 people, give it a budget. And you can see over here it's working through It's working through, it looks like my model wasn't set up. But normally you would see your events flow through here. I can see my traces and I can see my tools as they were being executed through all these flows. And it would zoom in and take a look at all these agents as they run. And so that's a great way to execute your agents, to see what they're doing, and to inspect them as they're being built. A couple other features that we've added, I'll move back over to PowerPoint here. beyond just building agents in an agent framework. We've also added an execution environment built on top of durable functions. If you're familiar with Azure Functions, you can now easily deploy and run your agents on top of those if you want a distributed system that you can use for building your agents on top of. And we've also added purview integration. So very easy for you to take your inputs and your outputs. We have like a middleware system with an agent framework that you connect up to, that's connected up to the purview API. So if you're using Microsoft purview for data loss prevention and analysis of your organization, of data in your organization, you can now connect that directly up to your agents and you can have your agents. use the same data loss prevention policies that you use within your organization. That's a very cool piece and very easy to do with the new Purview APIs. I'll pop back over to code real quick. I have another visual demo to talk about some of the UI pieces that we've added. We've added integration into both ChatKit and AGUI. So ChatKit is the UI library that OpenAI has developed for integrating with agents and agent systems. And it's a really nice set of UI packages that enable you to have a nice, clean, simple interface and customizable for your users. So you can see here, I have a demo here of ChatKit plus Agent Framework. And it kind of looks a little bit like ChatGPT. And that's kind of intentional. It gives you a familiar interface for your users. They have all the usual things. I can see my history. I can chat. I have a couple suggestions here. So I built this app that uses a tool. an MCP tool that goes and gets the weather. I can get the weather in New York. I can select a city here. One of the nice things with ChatKid is they have this concept of widgets so it can return a UI that the LLM can specify itself and then dynamically populate it with things like here I can see what the weather in Seattle is. Spoiler alert, the weather's not great in Seattle today. But as we get the results, okay, 29 degrees. Actually, that's not the right weather. I think that's supposed to be Fahrenheit. I think that messed up. It's cold and foggy here today. But we can also do that. We can also do other things like analyze an image. You can see here we can upload attachments. So, I can take an image here. I'll take a nice image of, I have this nice soda can, this agent soda can I generated earlier. So, we'll see if I can figure out what this is. But as all those controls makes it a lot easier for building in capabilities like file attachments. So, you can see here it's analyzed that image and it uses the model directly. As well as ChatKit, we built in support for AGUI. which is a protocol used a lot with UI components like CopilotKit in order to build UI on top of agent and AI systems. And very similar to ChatKit, you can build things, you can build a demo here. Here's a very simple, clean interface for using CopilotKit to build your agents. We can ask it to generate a sonnet for it. it'll work and we can do thumbs up and thumbs down and all those events can be returned to your application and copy paste. You can also do very complex things with agent framework I think with agent framework and AGUI. So there's a generating UI you know just like just like with CopilotKit you can generate dynamic UI and then what's really nice is you can share state between your agent and UI as well so you can build these complex forms that talk to your agent and that state is maintained by your agent. Okay, so that's some of the new tools in Agent Framework and some of the things that you can do when you're building agent systems with Agent Framework. But now that you've built your agent, let's talk about how you can host it within Foundry. So, if you built your agent in code, now you want to host it somewhere. And of course, you can host it on things I mentioned, durable functions. You can host it in Azure Container Apps. You can host it anywhere you want. You can host it in any kind of cloud service that you know, supports web services. It's just code in a framework. But we want to make it really easy for you to build your agents and your agent systems and then get them automatically and easily connected up to Foundry so that you can do things like evaluations and channels and monitoring of your agents without having to hook that up independently. We've enabled Foundry to host agents within the Foundry agent service. This allows you to take your custom code built in either Microsoft agent framework or land graph and with our developer tooling we'll package that up into a container and then deploy it and host it into Foundry. This gives you a managed runtime that's serverless and auto scales the compute and a dedicated execution environment. This is all based on Azure Container apps. You can integrate with the built-in tools in Foundry. So all the tools that you have deployed into Foundry, your agent tools, you can access those from your hosted agents. Of course, you can observe and monitor your agent execution. You can look at your performance metrics, your debug logs. You can do a cost analysis within Foundry. You can do continuous evaluation of your agent systems. You can see how they're performing over time. You can use the guardrails that are built into Foundry and the compliance checks. And you can see all your agents in the Foundry control plane. You can also interoperate between your agents and Foundry once they're deployed into Foundry as code agents, between Foundry and Copilot Studio or the Microsoft 365 or Agent 365 ecosystem. And of course you get all the foundation of Foundry like bring your own virtual networks and resources for state storage. You can use all those within hosted agents. So I'll flip over, I'll do a really quick demo of this. One of the things I'd like to show is that obviously I have I have a favorite around using agent framework, but we've enabled you to use the frameworks that you like. So for example, you can use this example here, which is, sorry, wrong page, this one here, which is an example using Langraph. So you can see I have my Langraph application here. This is a very simple sample from Langraph where they have, sorry, this sample is, Here we go. Okay, here's our LandGraph samples. This is a very simple calculator example from LandGraph. If you've ever used LandGraph, you probably started with this sample. It has a bunch of tools to enable you to do some basic math. And we can take this sample and once we've connected it up to Foundry using this agent.yaml to configure it. So we have this agent.yaml file which tells us about the agent. tells us the models it's using, tells us the protocol it's using like responses. We then use that and we use the tools you're already familiar with like AZD. We can call AZD AI agent create and we create this agent, we connect it into our tools and then we can do something like AZD up. So I have over here a line graph application. This is one of the samples from line graph. for building a calculator agent. Very simple, so it's helpful to assist in perform arithmetic. We've given it some tools to do addition and division and multiplication. This is one of the basic LandGraph samples. We can... We can then use this and deploy it and run it locally, debug it locally, execute it locally, do all the things that we do comfortably in our VS Code environment, use all the tools that we're familiar with for building applications. And then once we build it, we can use AZD to attach it to Foundry and then deploy it up to Foundry. So I was showing earlier the agent YAML file. This is our configuration file. that tells us how to connect it to Foundry, what are the environment variables and tools it needs to use and need to be specified in Foundry. So once I've selected my model and connected those to my models that are deployed in Foundry, also I can put my tools here if I had any custom Foundry tools that I was using. I can then use tools like AZD AI Agent, which I've already done here to create this agent and all the infrastructure and bicep files needed to deploy it. And then from there I can very easily do do AZD up and deploy this up into Foundry. This takes a few minutes to do so I'll pop over here to one I've deployed earlier over in Foundry. We can see this is the new Foundry portal. I have a few agents that I've created here but you can see here that calculator has been deployed and one of the things you can see is I have two different types of agents available now in Foundry. I have my declarative agents, my prompt agents that I'm familiar with where I can specify a prompt and tools and connections. within Foundry and test these and evaluate them and trace them and monitor them in Foundry. But I also have my code agents and my hosted agents. These are my agents that have been deployed into Foundry from my tooling. So this is the calculator agent I built earlier. I've deployed into Foundry. I can't edit the prompt now because it's all based in code, but I can still talk to it. I can... I can look at the traces and monitors and evaluations of this agent and I can see how it's performing over time. So this is a very simple agent, not that much to see, but I can look at the traces for it. I can look how often it's been run. I can monitor it. I can do evaluations. I can do all those things within the agent playground. I can also connect my agents together. So you saw I had two agents here. I had my calculator agent. I also had a problem interpreter agent, which is kind of takes natural language. breaks the problem down so it can be used by the calculator agent. And I can connect these together into a workflow. So just like I showed you workflows built in code and agent framework, we can also build these workflows visually in Foundry. So I can take agents, I can connect them together here. I have created these two agents. Actually, let's start this from the beginning. I can take two agents here, I've created this agent, I've connected it to the agent I've already created, which are my problem interpreter agent, and my code agent. So I can connect my LangGraph agents together with my agent framework agents together with my Foundry agents and have these all work together. So the sample I showed earlier with all those agents coordinating together, I could have easily built this visually right here in Foundry. And then we can evaluate it here by running it in preview. And we can say, you know, in natural language here, what is 7 times 6. And our first agent will interpret that in natural language and then generate the math problem that our calculator agent will now go and solve. So you can see the output of one agent is now becoming the input of the other agent. We can connect those together. We can build very complex workflows this way. We can also, you know, take these workflows that we've built in Foundry and we can migrate them back. into agent framework as well. So we can take the declarative version of these workflows, we can take the YAML, we can execute that directly into agent framework declaratively, or we can open it in VS Code and we can do a conversion there with that migration assistant from declarative foundry workflows into agent framework code in C-sharp or Python. Okay, so to recap, back to slides. We talked about a bunch of problems and challenges that developers have faced when they're building their multi-agent systems, and I talked about some solutions of these. So, where do I start to build an agent? Well, in my opinion, Microsoft Agent Framework is the way to go. We like all frameworks, though, at Foundry, and we've offered abilities and connectors to use all the major frameworks for building agent systems with Microsoft Foundry. We have a ton of new features in Microsoft Agent Framework, and we're really proud of the work that we've done there. You can connect to, you know, how do I connect my agents to other services? Well, we have full support for MCP and A2A for connecting both tools and agents together. How do I make my agents work together? We have dozens of patterns and samples and examples and workflows in the Microsoft Agent Framework repo that you can use to build these very sophisticated and reliable workflows for orchestrating not just your agents together, but even your code-based workflows, any sort of code you can orchestrate together and deploy at scale in a distributed system using Microsoft Agent Framework and things like the Durable Agent Framework. You can connect these into your telemetry systems using our OTEL telemetry as well as our new integration with Microsoft Purview so you can have your own DLP and compliance policies associated to your agent systems. And of course once you've built all your systems locally and have them running the way you want in your development environment, you can deploy those directly to Foundry whether you're using Microsoft Agent Framework or LandGraph. And then once you've built your agent systems, of course you want customers to use them. So we built integration with these really great user interface libraries from OpenAI's chat kit to AGUI with a lot of extensibility and capabilities already built into them for building really nice UIs on top of your agent systems. So just to wrap everything up, Foundry Agent Service, Microsoft Agent Framework, working together to allow you to build. your agent systems, connect them to your models, your Foundry IQ and your tools, building multi-agent systems, hosted agents and workflows together that you can now deploy into your own applications, into M365 and into Agent 365. All built on top of the foundational layers of security compliance and governance from Microsoft Foundry. If you want to learn more you can start today by typing ai.azure.com and trying out the new Foundry portal. The agent framework is open source and on GitHub. Please go try it out there. You search for Microsoft agent framework on GitHub and you'll find us. The team is all there responding to discussions and issues and you can join the community there. You can learn about building agents on Microsoft Learn. The QR code here you can try as well as engage directly with us through Discord or through our GitHub discussions on Foundry. Thank you and please enjoy the next year. I'm really excited to see what everyone can build with Microsoft Foundry and Microsoft Agent Framework.

Build multi-agent systems the right way with Microsoft Foundry. Go from single-agent prototypes to fleet-level orchestration using the Microsoft Foundry Agent Framework (Semantic Kernel + AutoGen), shared state, Human in the loop, OpenTel, MCP toolchains, A2A, and the Activity Protocol. Bring frameworks like LangGraph and OpenAI Agents SDK, then deploy as containerized, governed, observable agents on Microsoft Foundry.

Join the conversation on the Microsoft Foundry Discord: https://aka.ms/AIDevDays/Discord
Sign up for an Azure Free Trial: https://aka.ms/AzureFreeTrialYT
Get hands-on with today's MCP lab repo: https://aka.ms/AIDevDays/LabDay1
Catch up on-demand: https://aka.ms/AIDevDays/OnDemand

Chapters
00:00 – Introduction: Building AI Automation with Foundry
00:30 – Looking Back: A Year of AI Agents and Orchestration
01:43 – Developer Challenges: Where to Start and Common Questions
02:50 – Introducing Microsoft Agent Framework: Unified Experience
03:38 – Connecting to AI Services and Ecosystem Integration
05:11 – New UI Capabilities: ChatKit and AGUI for Agent Interfaces
06:09 – Demo: Creating Multi-Agent Workflows in VS Code
09:04 – Visualizing Agents with Dev UI and Workflow Builder
10:08 – Hosting Agents: Durable Functions and Purview Integration
11:03 – UI Enhancements: ChatKit Widgets and Dynamic Interfaces
12:59 – AGUI Integration: Building Rich Agent Experiences
14:02 – Hosting in Foundry: Managed Runtime and Observability
17:02 – Deploying LangGraph Agents to Foundry
19:27 – Foundry Portal: Managing Declarative and Code Agents
20:58 – Visual Workflow Builder in Foundry
22:27 – Recap: Solutions for Developer Challenges
24:19 – Getting Started: Resources and Community Links