From Chaos to Choreography: Multi-Agent Orchestration Patterns That Actually Work — Sandipan Bhaumik

Hi everyone, I'm Sandy. I've spent 18 years building data systems, a major part of it focusing on building and scaling distributed data systems in the cloud. I've done it for multi-tenant systems for software and SaaS companies, and then for scaling data and AI platforms in regulated industries like financial services and healthcare. I've learned a great deal about production-grade distributed systems while I have been working at AWS and now in Databricks. For the last two years, I've been deploying multi-agent AI systems in production and I have watched brilliant engineers make the same mistakes over and over. They think adding more agents is just like adding more features. It's not. It's building a distributed system. And today I'm going to show you the patterns that actually work when you make the transition. These are lessons that I have learned working in the trenches. And today I'm here to share it with you. Here's what we are covering today. First, the problem. I'll share you a very basic production world story about race conditions and why complexity of explodes when you go from one agent to five agents. I'll talk about the patterns, choreography and orchestration patterns for coordination of agents. I'll talk about state management, talk about failure recovery and how we can design for failure in production systems. And then I'll share how a production grade architecture will look like in a simple way possible. And I'll also show you an example on how we build this on Databricks. So let's dive into it. You see, one agent works beautifully. You have got your LLM, some prompts, maybe a retrieval augmented generation pipeline, maybe some tool calls. It demos great. Leadership loves it. You feel happy and your team is happy. And then product comes back. with a request that changes everything. They want five more agents and here's what happens. You think, okay, I know how to build agents and I will add five more. Except now you have coordination problems. Agent A produces data that Agent B needs. Agent C is waiting on both Agent A and Agent B. Agent D just updated the shared state that Agent B was reading and Agent E just crashed and took down this entire workflow. This is no longer an AI problem. This is a distributed system problem. And most of you didn't sign up to be distributed systems engineer. Let me tell you about a production deployment where this went very wrong. We built a credit decisioning system for a financial services company. The first agent credit score calculation worked perfectly. It worked great in demos, two weeks in production, zero issues. Then we added four more agents. income verification, risk assessment, fraud detection, and final approval. We deployed all five. In three days' time, we started seeing weird approvals. 20% of the decisions had incorrect risk ratings. Customers who should have been flagged were getting approved. The business team was panicking. It took us two days to find out what was happening. Credit score agent calculated a score of 750 and wrote to the database. The risk assessment agent, on the other hand, read from the database 500 milliseconds later and got a score of 680 for the same customer. Why did it happen? Because we had a caching layer for customer records. The right to PostgreSQL succeeded, but the cache was not invalidated. The risk agent read from the cache and it got stale data. It used the wrong score and made the wrong decision. This is a classic distributive systems problem. We had caching layer between the agents and the database. Cache invalidation failed and the agent was reading stale values. The race condition wasn't in the database. It was in the architecture. Multiple agents shared cache, no coordination on cache invalidation. This took us quite a while to find the pattern. It created delays in delivery and led to wrong decisions. And here's the lesson we learned. The problem was, of course, not with the model. The problem wasn't with the prompts. The problem was we built a distributed system without distributed system thinking. And that's what kills multi-agent projects. Not bad AI, but bad architecture. Now, I will show you the architecture that works. We will also look into a production-grade architecture. First, let's understand why this complexity explodes so quickly. Now, when you move from a one agent system to a multi agent, let's say five agent systems, it doesn't get just five times harder. It gets 25 times more complex. Coordination complexity grows exponentially. One agent has got zero coordination problems. Two agents have got at least one connection. Five agents have got at least 10 potential connections and coordination. Each connection is a failure point, a race condition, a state synchronization problem. You are not just building five agents. You are building a coordination problem across multiple relationships and across and possibility to have multiple failure modes. And that's why the complexity increases very, very quickly. Now I'm going to show you two critical patterns. First pattern is about how to coordinate multiple agents. Then we will talk about how you can manage state and then we'll talk about how we can recover and design for failure. Now, these patterns come from multiple years of distributed systems work and I can directly apply them on multi-agent AI system. Once you get the basics, it's really hard to miss these patterns when you build multi-agent AI architecture. The first decision you need to make is about choreography or orchestration. These are the two fundamental patterns for distributed coordination. Choreography means Agents coordinate through events. They are decentralized. They are autonomous. Orchestration means a central coordinator manages the workflow. This is centralized and controlled. Most teams pick one instinctively and regret it. Let me show you when to use each. Let's start with choreography. Choreography is event-driven. The research agent finishes research and publishes a research-completed event to a message bus. subscribe to that message bus and listens for that event type it is interested in. The analysis agent subscribes to that event type, picks it up, does analysis and publishes analysis ready. Then the report agent picks that analysis ready event, generates the report. There is no central coordinator here. Each agent is autonomous, listening for events it cares about, publishing when it is done. This is the beauty of choreography. Agents are loosely coupled. It's easy to add new agents and make them subscribe to the events that they're interested in. This drives high autonomy and scales really well. However, the nightmare of choreography is debugging. When something fails, you're playing detective with no real clue. Which agent failed to publish? Did the event get consumed? Did the event get consumed twice? You need bulletproof observability to make choreography work. Even with the event propagation, you need strong guarantees across delivery of these events. Without this, debugging is really hard. So when should you use choreography? You use choreography when your workflow is naturally event driven, when agents need to operate independently, when you're adding agents frequently and don't want to update a central coordinator. But it is important to understand. It is possible only if you have strong observability. If you can't trace events through your system, choreography will destroy you. I've seen teams choose choreography because it feels more agentic, more autonomous. Then they spend months firefighting because they can't debug distributed event flows. Don't make that mistake. Now let's look at the alternative. Orchestration. Orchestration is centralized. You have a workflow orchestrator that calls each agent directly. Agent A runs first, the orchestrator calls agent A, waits for the result, gets the result back, then the orchestrator calls agent B and C in parallel if there are agents that need to run in parallel. The orchestrator manages the parallelism, not the agents. B and C return their results to the orchestrator. Then the orchestrator calls agent D with the combined results from B and C. Every call goes through the orchestrator. Agents never call each other. The orchestrator is the single source of truth. It knows the entire execution graph. It manages state. It handles retries. It logs every step. Agents are dumb. They just take the input. They do the work. They return the output. The orchestrator does all the smart coordination. In Databricks, one way to implement this pattern would be with lang graph wired into AI agent framework as the orchestrator. But any workflow that gives you DAGs, direct acyclic graphs, and proper retry mechanisms would fit in this kind of orchestrator patterns. You use orchestration when you have complex dependencies that need central management, when you need to roll back, compensate for failures, when you want one dashboard showing the entire system state, when your workflow is relatively stable. In financial services, for example, we use orchestration almost exclusively. Why? because it provides easy debugging and the ability to roll back and that matters more than autonomy in this kind of industries. When something goes wrong with a credit decision, for example, we need to know exactly which agent made that call, in what order and with what data. Orchestration gives us that. Choreography doesn't. So how do you choose? Here's your decision framework. Two axes. Workflow complexity. simple to complex, autonomy requirements, low to high, simple workflow, high autonomy, you go with choreography. You need complex workflow with low autonomy tolerance, you go with orchestration. The interesting quadrant is the top right where you need complex workflow, but agents need autonomy. This is where you use hybrid patterns, choreography with saga patterns for compensation. I'll talk about this pattern later in this session as well. tools like Asian bricks on Databricks are starting to package these orchestration patterns for common multi-agent use cases. So you don't need to rebuild them every time. It makes building these patterns really easy in production environments. Now I use the decision metrics every time to make decisions with customers based on their use cases. It's worth you take a screenshot. I'm sure you will reference it. Let me show you what a production orchestration actually looks like at the tail end of this session. All right, now we have chosen a coordination pattern. Now let's talk about the thing that actually breaks when you scale. State. How do agents share data without race conditions, without stale reads, without mystery bugs? Here's what most people do first, and it's wrong. Shared mutable stage. Multiple agents writing at the same database records at the same time. Agent A reads credit score, calculates the value, writes it back. Agent B does the same thing at the same time. Both read 680. Agent A writes 750. Agent B writes 720. Last write means Agent A's update disappears. Lost update. I understand, yes, modern databases have protections in place, row logs, isolation levels, etc. But you have to use them correctly. Explicit transactions. You have to build serializable isolation. You have to make sure that you select for update. And many teams don't. They use default isolation. They don't use explicit locks. And they ship race condition to production. We did it. We did that mistake. And that resulted in delayed value to the business. We just assumed that the database would handle these conditions, but they don't. When it gets really complex, you have to handle them explicitly in the code. Now here's what works. Immutable state snapshots with versioning. Agent A produces a state version. Let's say version 1. It's sealed. It's immutable. Nobody can modify it. State is stored in the orchestrator database as an append-only log. These are insert operations, not any update. Agent A hands state version 1 to Agent B. Agent B validates the schema, checks that the data contract matches with its expectations. It processes it, produces state version 2, also immutable. Agent B inserts version 2 as the new row. It doesn't update version 1 and then hands it to Agent C. Same thing, schema validation, version tracking, immutability guarantee at each handoff. Agency fails. Now if agency fails, you roll back to version 2. If you need to debug, you replace state evolution from version 1 through version n. You can see exactly what each agent received and produced. This eliminates race conditions. No concurrent modification to the same record. Each agent appends a new version instead of updating the shared state. Now, of course, if you want to save these state snapshots, they can be logged. in any sort of append only storage for audit replay, but they are never shared for read or write. Now here's how it looks like in code. Agent state class, the frozen means immutable in Python. It has a version number, the data payload and who created it. The handoff function does three things. First, it validates the schema. This is the contract enforcement. We are checking that agent A's output matches agent B's input contract. This is critical and we will come back to this. Second, increment version. Create a new immutable state object with version n plus one. Third, execute the next agent with that immutable state. The agent can't modify the input state. It can only produce a new state. This prevents an entire class of bugs. It prevents race conditions on shared state. No stale rates. It provides a clear lineage. Every state has a version. and you know who has created it. When something goes wrong, you can trace back through state evolution. Version 7 produced bad output. Look into version 6 that went into the agent. Look at version 5 before that. You can binary search through your state history to find where things went wrong. And this becomes really, really powerful. Now, state management is half the battle. Data contracts are the other half. Agent A can just throw arbitrary data at agent B and hope it works. This doesn't work that way. They need a contract in place. In this example, research agent promises to output findings, confidence score, sources, timestamp, etc. Analysis agent declares it requires research agent output with type and first. And it validates if confidence is below 0.7, it will reject the handoff. This is the contract. If the research agent tries to hand off low quality data, the contract catches it at the boundary. You find out immediately, not three agents down the stream when it produces a report in garbage. When we work with our customers using Databricks, one way of doing it is registering these input output schemas in Unity catalog. So every agent's contract is versioned and governed in one place. All right, we talked about coordination. patterns we talked about state management now talk about now now let's talk about another thing that you need to keep in mind and that's failure and recovery and and the reason this is important is because agents will fail that's inevitable the llm will time out the api will rate limit you the agent will crash mid workflow what happens then what happens then is what you need to plan for and design in the system let's talk about few patterns Let's talk about the first pattern, which is a circuit breaker pattern. And this comes straight from the distributed system. When agent A calls agent B, it wraps that call in a circuit breaker. If agent B fails repeatedly, say five times in a row, the circuit breaker opens. Now, instead of waiting for a timeout every single time, you basically fail fast. Circuit open, agent B is down, you just try again later. You are not bombarding agent B with requests. you're protecting your system. After a timeout period, let's say 60 seconds, the circuit goes half open. Then you test agent B again with one request. If it succeeds, the circuit closes and normal operation resumes. If it fails, the circuit opens again and it resets the timer. This prevents you from cascading failures into the system. One agent going down doesn't bring your entire workflow down. You gracefully degrade. Maybe you skip that agent and continue with reduced functionality. Maybe you use cast results. Maybe you alert a human, but you don't crash the entire workflow. Circuit breakers are the single most important failure recovery pattern for multi-agent systems. Every agent call should be wrapped with the circuit breaker. We enforce the circuit breaker policies at the serving layer on Databricks through model serving or through AI gateway. Here's how it looks like in code. You track the failure count and you track the state. When you call an agent, you check the state first. If it is open, you fail first. You don't even try. If it is closed, you make the call. If the call succeeds, you reset the failure count and stay closed. If it fails, you increment the failure count. If you hit the threshold, you open the circuit. After the timeout period, you transition to half open. You test one request. If it succeeds, you close the circuit. If it fails, you open it again. This is a simple pattern, but it has got a massive impact. And in Databricks, you can log every open closed transition in MLflow. So you can see when an agent started flaking out. Now let's talk about another pattern. We call it the compensation pattern, also called saga pattern. Every agent has two methods, execute and compensate. Execute does the work, compensate rolls it back, undoes it. The orchestrator tracks which agents have executed. If the execution agent fails, the orchestrator walks backward through the executed agents and it calls compensate for each one. Analysis agent compensates, it deletes the draft recommendation from the system that it has written originally. And then the research agent compensates by clearing the cached research data that it gathered previously. So you're back to the initial state. No partial transactions, no stuck workflows. This is a simple rollback pattern that you can implement in multi-agent system. Compensation gives you transactional semantics across distributed agents. It is not sexy, but it's how production systems handle partial failures. Every orchestrated workflow needs this kind of compensation pattern, and you need to plan for it depending on what you're doing with your workflows. Here's how compensation looks in code. Every agent, as I mentioned earlier, has got two methods. The execution method. and the compensate method. The execution does the work, the compensate undoes it. That's the contract. Every operation must be reversible. The orchestrator tracks which agents have run successfully and then it keeps the list. Agent A executes, gets added. Agent B executes, gets added. Agent C fails. Now we walk backward through the list in reverse order. Agent B compensates first, it undoes the work that it has done. Agent A compensates next, it undoes the work that Agent A has done, and it goes back to the initial state. This is saga pattern from distributed databases. Financial services requires this. Now that we have covered these different patterns, I wanted to show you what a production architecture would look like when you bring these things together. You have got the orchestrator at the left hand side. It's the brain of the workflow. it contains the workflow engine, it contains the state store holding versions to 0 to n and it can look into the observability layer. It handles the observability data. Every call goes to the orchestrator. Orchestrator calls agent A. Agent A returns state version 1 to the orchestrator. Orchestrator then calls agent B and C in parallel if they need to run in parallel. Both receives state version 1 from the orchestrator. They return results. Orchestrator stores at version 2 and 3. Finally, orchestrator calls D with these combined results. Agents never call each other. All coordination happens through the orchestrator. And this is what gives us control, observability, capability to roll back. This runs 24 x 7 across billions of transactions because the orchestrator is the single source of truth. All right. Here's a production architecture. that you could implement with the Databricks Data Intelligence platform. In the orchestration layer, you can have Langraph wired into Mosaic AI agent framework. It handles multi-agent orchestration. It manages the workflow graph and knows which agents to call in what order. Each agent is implemented as a Unity Catalog function. It could be written in SQL or Python, or it could be a model registered in a Unity Catalog. They are, when you register these assets in Unity Catalog, they are discoverable centrally within the organization. They can be governed in one place and they can be versioned, which is really critical in terms of operating these workflows in production. We expose these agents through a database model serving or function serving, and that's where we enforce this circuit breaker style policies like retries or timeouts or rate limits. at the serving layer, typically via AI Gateway configuration. Now, when we talk about the data layer, Delta Lake stores everything. It not only stores the state versions from the agent, it also stores customer data and, you know, all the data that you need for your workflows to work. Talking about the state snapshots, Delta Table is immutable and versioned. For us, those state versions are just rows in a delta table. We never update them in place. Each agent run is tied to a state version via MLflow traces so we can step through the evolution when something breaks. Now, I just wanted to touch upon Unity Catalog. It governs everything, access control, lineage, audit trail for both data and agents. MLflow gives us per agent tracing evaluation capabilities with out of the box LLM as judges and metrics on every call. And as I mentioned earlier, tools like Agent Bricks is the higher level way of Databricks packaging these orchestration patterns for common multi-agent use cases. So you don't need to rebuild them every time. So just to wrap up this workflow, you see the Landgraf orchestrator calls Agent A a Unity Catalog function or model. It gets the result. writes version one state to delta it then calls agent b with state version one writes version two and so on ml flow traces every call latency inputs outputs token usage a circuit breaker at the serving layer guards each call if agency fails langrath triggers compensation logic and walks backward calling the compensate functions for previous successful steps. These kind of patterns run in production day in and day out. So thank you for hearing me out. You can reach out to me over LinkedIn. You can scan this keyword that will take you directly to my LinkedIn profile. I would like to leave you with three final thoughts. First of all, agent chaos is inevitable. When you scale past one agent, you will hit coordination problems, race conditions. cascading failures. That's guaranteed. The complexity curve doesn't lie. Agent choreography is a choice. You can build systems with proper patterns, orchestration, choreography, immutable state, circuit breakers, compensation patterns, data contracts. Make sure you understand these patterns and bring them to your production architecture. Doing so will help you build systems, not demos. Demos are easy. You use an LLM to show something cool. Everyone can do it. These things don't work in production. In production, you have to build systems and systems are hard. Systems are what create value for businesses. Everything I showed you today, choreography versus orchestration, immutable state, circuit breakers, these are all unsexy infrastructure work. You won't get applause for implementing a circuit breaker, but you make your systems more reliable. They don't fail at 2 a.m. in the night. That is what people notice over time. Be a systems engineer. The patterns here, they work. Apply these patterns in your production architecture. Thank you very much for watching. Bye.

One AI agent is a feature. Fifty agents is a distributed systems problem nobody's discussing. I've seen this pattern: teams build one agent, then five, then drown in coordination problems unrelated to LLMs. Agent handoffs fail silently. Data goes stale. Decisions become untraceable. Drawing from Databricks production deployments, I'll expose orchestration anti-patterns killing multi-agent systems and show agent handoff protocols that work—state management, data contracts, failure modes. You'll see when to choreograph versus orchestrate and live multi-agent workflow with proper observability. This applies distributed systems engineering to agents: the infrastructure layer everyone needs but nobody's building.

Sandipan Bhaumik - Data & AI Tech Lead, Databricks

Sandipan Bhaumik has spent 18 years building data and AI systems inside environments that can't afford them to fail - NHS, Tier 1 banks, and large enterprises across EMEA. At AWS and now Databricks, he's seen firsthand where multi-agent systems break down between architecture and production. He is a regular speaker on data and AI system architecutr ebest practices, runs a community of AI practitioners, and he's here to talk about what actually holds together when you scale agentic AI systems in production.

Socials:
https://www.linkedin.com/in/sandipanbhaumik

Slides:
https://drive.google.com/file/d/18LqVzhfVS3iULYuy2EshWoMLmQt3rdpT/view?usp=sharing