Capital One @ Replay 2026 | Supercharging Cloud Orchestration with Temporal Open Source
Temporal · 2026-06-12 · 39м 46с · 416 просмотров · YouTube ↗
Топики: durable-execution
🎧 Аудио
📝 Summary
model=deepseek-v4-flash · prompt=summary-v7 · 9 037→3 297 tokens · 2026-07-20 14:57:00
🎯 Главная суть
Capital One перевела облачное provisioning инфраструктуры с ручного IaC и event-driven choreography на централизованную orchestration с Temporal Open Source. За 2024 год они развернули self-hosted кластер на Fargate + RDS, внедрили multi-tenancy для десятков команд, стандартизировали approval-процессы и сократили время доставки новых возможностей с месяцев до дней, попутно получив полную наблюдаемость и аудируемость всех workflow.
Эволюция provisioning: три поколения
В 2015 году Capital One решила полностью перейти на AWS — для банка с жёстким регулированием это был радикальный шаг. Это заставило заново определить, как развёртывать и управлять облачной инфраструктурой на масштабе предприятия. Первое поколение (Gen1) строилось на Infrastructure as Code (IaC) — Terraform-подобные инструменты давали версионирование состояния и теоретически лёгкий откат. На практике любые изменения вне пайплайна создавали дрейф (drift), из-за которого откаты превращались в ночные кошмары. Обработка ошибок становилась всё более громоздкой по мере роста сложности.
Второе поколение (Gen2) перешло от статического provisioning к динамическому: choreography на лямбдах с event-driven архитектурой. Это похоже на группу синхронных пловцов — каждый знает свою задачу, но нет центрального дирижёра. Система получила визуальные state-машины, упростилась логика, но отладка усложнилась, а управлять тысячами Lambda было огромной нагрузкой на разработчиков. Каждый сервис должен был понимать failure modes других. Откат требовал программирования обратного пути по happy path.
Третье поколение (Gen3) — централизованная orchestration с durable execution на Temporal. Переход позволил избавиться от недостатков choreography: теперь есть единый координатор (как дирижёр оркестра), который управляет разнородными компонентами, обеспечивает fault tolerance и долгоживущие процессы.
Почему Temporal Open Source, а не managed-сервис
Для Gen3 Capital One искала платформу, соответствующую их принципам: code-first (не YAML/JSON конфиги), reuse существующих API и кода из Gen1/Gen2, автоматическое управление историей workflow (continue-as-new для длинных процессов), полный audit trail с идентификацией. Temporal предоставила всё это из коробки. Команде не пришлось писать собственный механизм продолжения новых workflow при неограниченном росте event history — это встроено. Кроме того, immutable история каждого выполнения дала риск-офицерам банка прозрачность, к которой они привыкли в on-prem мире.
С 2024 года платформа выросла: 50+ инфраструктурных возможностей, 100+ участников, разработчики доставляют задачи, раньше занимавшие от двух недель до года, за дни или часы. Ещё один бонус — Convey’s law: команды перестали быть жёстко привязаны к продуктам; теперь они управляют workflow, и реорганизация не ведёт к потере поддержки.
Архитектура self-hosted Temporal кластера
Capital One развернула Open Source Temporal на AWS Fargate (контейнеры без управления EC2), выбрав простоту и снижение operational overhead. Кластер состоит из frontend (gateway), matching, history и worker-сервисов — все как Fargate-контейнеры за NLB. Для хранения данных использовали один RDS Postgres (две БД: temporal — состояние выполнения, visibility — поиск и фильтрация). Для динамических конфигураций — EFS mount. Архив завершённых workflow уходит в S3 bucket с собственными lifecycle-политиками для трейсабельности.
Для UI взяли оригинальный код Temporal (Go binary), который интегрируется с корпоративным SSO через OIDC. CLI использует API key с коротким TTL, для чего написали login wrapper для обновления токенов. Авторизация построена на JWT: кастомный authorizer читает claims и сопоставляет с namespace + роль (admin/writer/reader). Таким образом, как люди, так и машины проходят через единую OIDC-инфраструктуру Capital One, не изобретая велосипед.
Multi-tenancy: центральный кластер и распределённые команды
Каждая команда владеет своим кусочком облачного provisioning. Все их worker'ы живут в собственных AWS-аккаунтах, но подключаются к одному центральному Temporal-кластеру. Для упрощения онбординга Capital One разработала лёгкие SDK на нескольких языках, которые инкапсулируют авторизацию и подключение к OTEL (OpenTelemetry). На каждом worker стоит sidecar, собирающий метрики и отправляющий их в центральный OTEL gateway. Благодаря span-атрибутам, внедрённым SDK, центральная команда видит, кто и как использует кластер: кто нарушает SLA, вызывает несуществующие сущности, перегружает систему. Это баланс между свободой команд и централизованной видимостью.
Вызовы при развёртывании в жёстко регулируемой среде
Compliance Capital One добавил несколько ограничений: миграции БД должны следовать внутренним SDLC-процессам, что не всегда совпадает с тулингом Temporal. Пришлось потратить время на интеграцию. Также требовалось строгое разделение обязанностей (segregation of duties) — ни человек, ни машина не должны иметь несанкционированный доступ. Выбор Cassandra или Kubernetes был отвергнут из-за высокой compliance-нагрузки; остановились на Fargate + RDS как на простых и управляемых сервисах.
Самая сложная часть — настройка авторизации. Пришлось написать custom authorizer и claims mapper, а также модифицировать CLI для работы с короткоживущими API-ключами. Поддержка нескольких языков SDK добавляет overhead. Resiliency тоже потребовала внимания: Temporal реплицирует данные на compute-уровне, поэтому нужно было продумать, как организовать репликацию между кластерами. Тем не менее, все эти барьеры преодолимы при достаточном начальном инжиниринге.
Измеримые результаты после внедрения
Первое — операционная скорость: благодаря сигналам Temporal команды начали отслеживать manual-процессы, которые раньше были «чёрным ящиком». Теперь лидеры видят, сколько времени занимает каждый шаг (автоматический и ручной), и могут принимать решения по узким местам на основе данных. Второе — скорость доставки: workflow стали Lego-блоками, которые комбинируются под новые запросы за дни. Третье — сокращение дублирования (arbitrary uniqueness): раньше разные команды строили похожие database, UI или approval-процессы с уникальными особенностями; центральная платформа заставила их стандартизироваться, что снизило затраты на поддержку «снежинок».
Техническая надёжность выросла не только за счёт durable execution. Разработчики, работая с Temporal, вынуждены думать о сагах, компенсирующих транзакциях и откатах с первого дня — это меняет культуру в сторону более надёжного ПО. Наблюдаемость (observability) в multi-tenant архитектуре, где трудно искать ошибки между командами, радикально улучшилась: центральный кластер даёт сквозной проход по workflow, показывая, что сломалось и кто виноват.
Best practices и first principles для масштаба
На основе опыта Capital One сформулировала несколько жёстких уроков:
- Code-first, not config: activities определяются как обычные функции на Go/Python/TypeScript; никаких YAML-глыб. Temporal позволяет обернуть существующие API в activities без переписывания (но иногда код всё же надо переписать из-за неидепотентности или недетерминизма).
- Workflows — для принятия решений, activities — для выполнения: весь тяжёлый код и бизнес-логика должны быть в activities. Workflow должен быть детерминирован: при одинаковом запросе всегда выдавать одинаковый результат. Если workflow падает, он восстанавливается только из activities; весь код вне activities теряется.
- Не добавлять retry-логику в activity: Temporal уже делает retry с at-least-once гарантией. Полагаться на это.
- Выбор таймаутов: schedule-to-close, start-to-close и другие нужно стандартизировать в рамках платформы, иначе каждая команда выбирает по-своему, что ведёт к хаосу.
- Версионирование и heartbeats: для short-lived workflow — не нужны, для long-lived (месяцы/годы) — обязательны. Плюс continue-as-new для контроля размера event history.
- Общие workflow как сервис: вместо того чтобы 10 команд писали свои Slack-уведомления, Capital One выложила common namespace с общими workers (открытие production change, дашборды и т.п.), которыми пользуются все.
Обучение разработчиков оказалось критичным: они по умолчанию не думают в парадигме durable execution, поэтому Temporal провела тренинги. Также создали code review checklist для единообразного использования сигналов, update, heartbeat и таймаутов.
Когда выбирать orchestration, а когда choreography
Vivek резюмирует: если задача простая — choreography (Lambda, event-driven) может быть достаточной и orchestration будет overkill. Но для сложных, долгоживущих процессов с требованиями к откату, аудиту и наблюдаемости orchestration на Temporal даёт преимущество. Также надо решить: self-host или managed Temporal Cloud. Capital One выбрала self-host из-за compliance и контроля, но это потребовало значительных инженерных усилий по авторизации, SDLC и resilience. В конечном счёте, платформа оказалась выгодной: «Всё ломается всё время. Durable execution позволяет быть к этому готовым».
📜 Transcript
en · 6 048 слов · 85 сегментов · flagged: word_run (1 dropped, q=0.98)
Показать текст транскрипта
Good afternoon everybody. Thank you for coming and I hope everyone's had a great week this week at Replay. Together we have two speakers for you today, Vivek Gupta and Joseph Gilbert from Capital One. Vivek is a distinguished engineer and director of software engineering at Capital One, leading enterprise-wide cloud modernization and overseeing Capital One's durable execution community of practice. He is joined by Joseph Gilbert. What a fun last couple of days at Replay. I'm sure I learned a lot, and I'm sure you guys learned a lot too. Welcome to our talk, the final talk of the day. It's a real production success story of Capital One, how we transformed our cloud infrastructure provisioning using temporal open source. We'll walk you through our implementation, our evolution, how we self-hosted Temporal. We'll talk about the challenges we faced and how we solved them. Finally, we'll have some key takeaways for you as you are also going through your journey at Temporal or with Temporal that may help you. All right. So I'm Vivek. I'm a distinguished engineer in Cloud Platforms team at Capital One. And I'm Joseph Gilbert. I'm the lead engineer for our Temporal integration. So our team not only manages and owns the platform, but also provisions and govern the Capital One's core cloud infrastructure provisioning at scale. And every application team depends on our platform. I also lead our durable execution community of practice, which I'm really excited about. It's an enterprise-wide forum where teams discuss orchestration solutions. and durable execution best practices, different other vendor models, and we have jumpstart each other with their POCs. All right, so I'm going to share a story that walks us through three generations of Capital One's cloud provisioning evolution. Each generation was a direct response to the requirements we had, to the opportunities and roadblocks we faced. The problem we faced are not unique just to us and are likely familiar to many in this room. So it all started in 2015 when Capital One made a bold move to go all in on AWS Cloud. For a highly regulated financial institution, this was way ahead of its time. That decision sort of forced us... to define the provisioning for Capital One. How do you stand up and govern cloud infrastructure at enterprise scale with compliance, with risk mitigation across a large footprint? So after that 2015 announcement is when a Gen 1 sort of really came about. Gen 1 is our first generation of how we build software. We realized the scale and demands of cloud. the vertical and horizontal scaling that we needed. And so in Gen 1, in this phase, we were predominantly using infrastructure as code. Major growth in our cloud footprint was the reason we evolved to a choreographed system in Gen 2. Think of choreography as a group of swimmers, synchronized swimmers. They are swimming, each swimmer know what to do and what their job is, but there's no centralized coordinator that is helping the swimmers. That was us in Gen 2. We moved from a model of static provisioning to dynamic provisioning. Today, we leverage durable execution and orchestrated workflows for our multi-tenant systems. So we went from dynamic provisioning to centralized orchestration. To understand orchestration, let's take an example of an orchestra. So when a conductor in an orchestra conducts and instructs each musician, they have very different capabilities, some playing violin, some playing cello and so on. But based on the instructions from the orchestra or from the conductor, the orchestra runs smooth and produce a beautiful symphony. Similarly, orchestration helped us handle different puzzle pieces, different Lego blocks together in our journey. I want to double click a little bit on our initial journey of infrastructure as code. We realized our provisioning models had to change from an on-prem pattern to now that we are in cloud in Gen 1. One of the best things about infrastructure as code at the time was version state. It gave us a state that was theoretically controlled and rollbacks were easy via state tracking. But the theory sounded great. In reality, it was difficult the operational changes that went outside of our code pipeline created drift and because of that drift it was really hard to do rollbacks i'm sure you are all familiar with 2am on call sessions right so that complexity of work as it increased the error handling became more cumbersome and then we moved to a model of static provisioning to dynamic provisioning in this centralized approach where services sort of work together without a centralized controller but exchanging messages gen 2 gave us event-driven lambda-based choreographed solutions which gave us visual state machines so no more hidden procedural logic in this model you know our services were easy to scale but hard to debug. They required well-defined message contracts. And honestly, managing thousands and thousands fleet of Lambda was not an easy task. It was an enormous burden on the developers. In the choreographed system in general, this is a problem where each service in the choreographed system need to understand the failure modes of the other systems that are taking in part. So rollback required happy. a path programming in reverse, which was cumbersome. Here we are now, the bridge to Gen 3. In Capital One, we always say crossing the canyon. Our CEO says that we have to cross the canyon from on-prem to cloud. So I thought this would be a fitting slide, how we cross the canyon from Gen 2 to Gen 3. We had a vision for platform provisioning. We needed a platform where inner sourcing is seamless and easy. Rollbacks are not an afterthought. systems that is fault tolerant resilient and can provide durability with simplified error handling so when we talk about durable execution that's when we discovered durable execution a term coined by temporal so what is durable execution it's a distributed systems ability to maintain application state and execution even in the middle of interruptions and failures it is a practice of making execution state persistent so that services can recover when the failures happen but they don't need to do the same steps again and again and they are idempotent in nature durable execution provided us a number of things that we were looking for a platform fault tolerant resilient long-running processes that need to go for not just days or weeks but months or years Now I'm going to hand off to Joseph to talk more about our customized deployment of open source temporal stack integrated within the Capital One infrastructure boundaries and controls. Joseph, take us away into the world of temporal, or I should be on theme and say the outer space of temporal. OK. All right. Thanks, Vivek. All right. So in order to be successful with our orchestration, we needed to move beyond the old way of how we had handled cloud provisioning in the past. And to that end, we had implemented a temporal open source architecture to build a multi-tenant system in our space in order to bring many teams together into a single unified deployment for cloud provisioning and orchestration. So today I'm going to walk you through some of the things on how we did that, some of the things that we built. We'll talk a little bit about some real world challenges that we had that you might face if you're going to be doing something similar. And also we'll talk a little bit about the values and impacts that we have seen and this provisioning process. So for our deployment of Temporal, we use a container-first serverless approach. We went with a simple solution, which is Fargate for us, that minimizes our operational overhead. It also reduces some complexity. We try to stay away from things that provide vulnerabilities to us. So this means for us no EC2 nodes that we need to manage. We did not go with Fargate in this solution, although that is a completely valid solution. But I do want to say that this was a good choice for both simplicity and cost. Now, as you know, the temporal cluster itself consists of four different main services. There's a front-end service. That acts as the gateway for all inbound traffic and there are four other services. There's a matching service a history and a worker service all of these are deployed into the cluster through Fargate They're deployed as containers and we front that with an LB which acts as an inbound for all the traffic into the cluster Now we experimented with an ALB and an LB both work fine. It's just up to your preference For our storage layer Temporal's open source distribution has two types of configuration files. There are dynamic and static configuration files. We used an EFS mount within those spaces in order to load the dynamic configuration files. They get polled and polled. As for our data layer, we wanted the simple RDS Postgres instance. Very simple, nothing too fancy. Part of this was driven by the fact that, and we'll talk about this in a minute, that temporal likes to do its replication at the compute layer rather than at the database layer. So using a simple database is perfect. Now within that database, there are two databases, because it's Postgres. There is a temporal database which manages the execution state and saga pattern. And there's a visibility database that we put in there as well to manage the searching, filtering, and other components. For archival, we're setting up an S3 buckets for archival. As workflows roll off the UI, they get sent off to S3 for retention. Within that space, within S3, we can all set up our own lifecycle policies for our own purposes, for our retention policies on those execution states that provide some traceability into what happened within our space. And on the security side, Our cluster is configured to handle authorization through a custom authorizer and claim mapper, which I'll talk a little bit more about. So whether it is a worker or a client connecting through that NLB, they're talking to that custom authorizer to connect to the cluster. So one last important thing to note here is that as a multi-tenant system, all of these workers are owned by different teams. At Capital One, we have a number of specialized teams which own part of the cloud provisioning process. And as such, they have their own access controls and isolation spaces. All these workers connect back into the central cluster and are able to connect in and we manage the central cluster. For each of these workers, we run a sidecar to handle open telemetry for OTEL. Temporal exports already a default set of metrics for their SDKs. For their core metrics, they export a certain set of metrics, and through SDKs, they send out worker metrics. We collect those through our sidecars and export them to an Elto gateway for collection so we can visualize them. For the core metrics, it helps us understand exactly how the core is performing. For the workers, it tells us what they're doing. So if you are looking to build out something similar in this space, this is a relatively simple and viable solution for building out or bootstrapping your own cluster. All right. So one of the core things we're trying to build with this target state architecture is a truly multi-tenant system. Now, with various teams and disparate ownership over different processes in our cloud provisioning process, our goal is to get these teams to orchestrate these activities seamlessly so that they can start delivering that single product for our internal customer base. So every team sends their traffic to the front end of the cluster, which we front with the NLB. But keep in mind that it's not just workers connecting to that front end. So you also have other clients connected in the front as well, such as the CLI and the UI also connect those clients into the front end of Temporal. For the UI, we took the Temporal code, or the UI code, which is embedded in a Go binary. We ran that, and that works great. The great thing about this is that it integrates with your on-prem SSO already, so it'll fit right into your organization's existing identity providers. Then we have the CLI, which allows teams to kick off workflows as well. Within the CLI, there is an API key feature that will allow you to authorize into your cluster through that same front-end endpoint. So to help the teams onboard, our teams to onboard into this multi-tenant cluster, we did develop a small set of SDKs to help teams onboard into the authorization and metrics platforms that we provide. So that helps teams be guided through this process so they don't have to guess on how to connect with our authorization or open telemetry systems. Now we mentioned that each one of these remote workers has an OTEL sidecar attached to it. Sidecars connect to a centralized hotel gateway to send back metrics. But one of the things that we didn't talk about is that in our SDKs, our lightweight SDKs that we provide for our internal customers, we can inject span attributes into those metrics and we can then leverage those spans to give our central team some visibility into what's happening on the workers in our multi-tenant architecture. So we know who's behaving well. We know who's calling things that don't exist. We know who might be throwing too much work at the cluster. Some teams just need help troubleshooting issues that they don't even realize that they have. And that's where we come in. So it's all about the balance. We're trying to give teams freedom to run in their own accounts, but provide the centralized tooling to help onboard them and provide them with the visibility tools that they need. All right, so let's switch gears a little bit and talk about authorization. When you're setting up a temporal cluster, you really have two choices. You're going to go with either MTLS or you're going to go with JWT. Both of these have the pros and cons, but for many open source teams, JWT is a natural fit within your organization. At Capital One, it's an obvious choice because we already have an SSO and OIDC infrastructure, so it makes sense to be able to integrate into that space. Now, there's nothing wrong with issuing MTLS certificates. But we found by going the JWT route, it's just a simpler onboarding process and it's smoother for our development teams. So inside of Temporal, the primary security boundary is the namespace. This is where you segregate workers and enforce your access control. Within the namespace are three main roles. There's an admin, there's a writer and a reader. I'm sure this is subject to change, but this is where we, when we deployed it, this was true. The permission space is essentially therefore the namespace plus the role. So when a worker connects, it presents its claims through the JWT and then Temporal's authorizer, custom authorizer that we have, reads those claims and says you have access to a namespace with a role. Now one thing to keep in mind when you're thinking about your authorization systems is that you're really authorizing two types of identities in your cluster. You're authorizing humans, and you're authorizing machines. So the humans might be using the CLI, and they might be using the UI, and the machines might be using the workers integrating into the cluster. And so both of these have authorization boundaries that you need to think about when you're onboarding and setting up your authorization. So this is how we did it. Both humans and workers connect to our OIDC platform. They pick up the required permissions there. They pass them back as a token. The token is passed back to the temporal system, which authorizes the claims and gives them access to the cluster. So by leveraging your existing identity layers, you don't have to reinvent the wheel here. You can integrate the existing systems, your OIDC systems, into for authorization. All right, so since we've now talked a little bit about the how, we're going to talk about some challenges that we faced in the integration, some hurdles. To be fair, none of these were showstoppers, but they might be things that you might be facing if you're trying to do something very similar in this regard. So a bit of context here. At Capital One. We operate under a strict set of compliance obligations. We have a lot of internal standards regarding how we deploy the software, what we can use, and just as importantly, what we can't use. So as I talk through these choices, keep them in mind. They may not be the default settings for everybody, but they are the right fit in our environment. So first of all, we need to talk about the SDLC processes. For us, deploying and configuring software might look a little different than the way that standard temporal might suggest. So for instance, a database migration might need to follow a specific internal process that doesn't always line up with the box with the existing tooling. That took us some time to figure out how to bridge that engineering gap. Another major focus for us is segregation of duties. We are required to ensure that no one, either humans or machines, have access to things that they're not approved for. So there is... Additionally, there are some choices of technology that might be powerful but carry high compliance burdens. So for example, something like Cassandra or Kubernetes could be an option, but they can be more challenging to secure in certain environments, which is one of the reasons that led us down the path to looking at Fargate and RDS. They're simple, they're well managed, it's a great starting point for you. At the end of the day, every tech choice from authorization boundaries to the deployment model is guided by our governance framework and it's the length through which we do business. All right, so one of the things that I want to call out first is specifically is one of the big challenges that you'll find in setting up self-hosted for yourself is working in getting authorization set up. It's probably one of the more challenging hurdles that we saw when we set ours up. It does require a significant amount of lifting and some thought. We did talk about the MTLS and JWT choices already, but there's some other gotchas in this space. So, for instance, the CLI is able to support the API key, but that API key might have a TTL on it of a short time frame. So, there's some work that you might need to do to sort through that process. Additionally, because we had to write some SDKs to help onboard, Our teams through the onboarding process we had to make some wrappers, but those wrappers are written in different languages So we are maintaining a couple different languages on what we support if you did something similar for your teams You might have to think about supporting a couple of lightweight SDKs for the languages that you support in addition to that We did need to do some work with a custom authorizer and the custom claims mapper so When you first start in this process and you're going to stand up your open source distribution, you need to think about a little bit about how you're handling the off end part of your connection, how are you handling the permissions boundaries, and where are they stored. All right, there's a couple other items that we had in the challenges and hurdles section. Go through them pretty quickly. One of them was resiliency. Temporal's natural replication model is to happen at the compute layer. For you, this means that you have to think about how you're going to provide that sort of replication in your environment, where you're going to put your replication in replicated clusters, and if you're able to support compute layer replication between those clusters for resiliency. We mentioned a little bit about the challenges with the CLI. We solved that problem by creating some sort of a login wrapper that allows us to refresh the tokens. And in our case, we did need to modify a little bit the way that we do the database migrations from using the normal internal tooling. So that being said, this is all absolutely doable. The temporal community has been incredibly helpful. The temporal team themselves have been right there with us working through these hurdles. And if you are willing to put in the initial engineering effort to bridge these gaps, you can end up with a powerful and resilient secure platform that can handle complex enterprise workflows. All right, so now that we've gone through some of these hurdles, I want to talk about some of the payoffs, which is great. Since moving to the self-host model, we've seen some really interesting aspects across the organization, some interesting impacts across the organization. Some of these are expected, some of them weren't. Let's talk about them. So one of the first major impacts that we've seen, which was a bit of a surprise to me, was the ability to provide a real boost to our operational velocity. And what that means, one of the ways that we can do that is start tracking manual processes. Every team has been on an automation journey. I'm sure you've been on an automation journey for years. One team automates a component, another team automates another component, and they're bridged together with some manual glue process. Right? Drive those. Temporal helps you drive those processes into temporal. use signals. When that manual process kicks off, drive it into a signal. When the manual process ends, drive that into the end of the signal. We can now see where all this time is going to. We can track exactly how long a job takes, both the automated parts and the human ones. It's a big game changer because if you're doing data-driven decision-making, leadership finally gets a clear in-the-box picture of where all the bottlenecks are in your processes. Secondly, we're delivering our automation much faster. What this means is in this multi-tenant system, these workflows are effectively Lego blocks. We pull them together to build new things together when new requests comes in. Asks are now delivered in days where they may have taken longer because we can cobble together pieces from multiple teams in order to handle and deliver. deliver components where before we may have taken a much longer time. And the last thing I want to talk about is the concept of arbitrary uniqueness. So we have seen a massive reduction in this process. You've seen it in your own teams. We've got one group over here building something, another team building something over here. There are slight variations on each other. Maybe it's a database, maybe it's a UI, maybe it's an approval process. But one of the things we have seen is by driving everyone into a central platform, we drive commonalities. And what this means is standardized workflows for things like approval processes or change management processes. And it doesn't just simplify your life, it helps you reduce your costs. Teams no longer have to maintain their independent, specialized snowflake infrastructures, but ultimately you're costing money. In the end, this isn't just about writing your code. It's about standardization and cost reduction. And ultimately, it's a better product for our customers. And finally, I want to talk about one of the other aspects we've seen with Temporal, which is its technical reliability. Now, we all know the selling point, durable execution. But beyond that, I want to talk about two other impacts that we've seen. One of them is the development of a developer culture which is a bit of a shift. So as engineers, we like to do happy path programming. We like to make it work. We like to get it done. But working in Temporal, you start thinking about sagas. You start thinking about compensating transactions. You start thinking about rollbacks. This is really forcing our developers to start thinking about unhappy paths from day one as they're building the code. It creates a positive development culture and it starts spawning conversations within organizations and within teams to think about how to deliver reliable software. And the other impact which I think is really understated is that Temporal provides a massive boost to observability. In a multi-tenor architecture, it can be really hard to track down exactly where something broke, who did what, tracing through logs, trying to figure hunting down issues between different teams. When all those teams are driven into a centralized structure, we're able to now walk through that code, see who did what, where the problem was. Engineers can jump in and troubleshoot those problems in real time. This level of visibility is amazing. This is really a big win for our observability on our platform, and it helps make our engineers more intentional and our entire delivery process more transparent. All right. Well, this concludes our deep dive into the what and the why of our journey. I want to turn it back over to Vivek, who helped take us through some key takeaways and best practices we've gathered while supercharging our cloud orchestration. VIVEK HANIRAJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJANIJ you know our development journey our evolution we have talked about you know what went wrong and what were the challenges how we fixed them now let's talk about some of the key takeaways i'm going to walk you through a set of our first principles these are the guiding principles they were not abstract whenever we had to make trade-offs in our architecture decisions we use these principles to think about one-way door decisions or two-way door decisions They're also what we would recommend if you are building on Temporal at scale. So why did we choose Temporal open source? Our platform was built using these first principles. Temporal provided certain capabilities that we were looking for. Code first, not config. In Temporal ecosystems, activities, for example, can be defined as normal functions. Workflows and activities are written in code of your choice. YAML state files to manage, no JSON blobs to configs to handle, workflows are Go, Python, TypeScript, et cetera, using the same tools and testing that the developers are already familiar with. The other important part here is that leveraging existing assets. We have built a lot of APIs. We have written a lot of code during our Gen 1 and Gen 2 days. So we wanted to reuse that code. Temporal provides certain primitives so that you could wrap the existing API or you could, you know, either you could wrap the existing API or SDK logic into activities. And no rewrites were required. But some APIs did require some rewrites because Temporal very soon exposed the weakness in the code when the code was, you know, not idempotent, when we were having determinism errors. So we had to rewrite some stuff, which I think it's a really good point. When you start using Temporal, you start thinking about the behavior of the code, almost. We also wanted to make sure that teams don't have to worry about everything, how the backward interoperability works, et cetera. They just need to maintain their code because that's what they are good at, right? Thinking about reliable execution, let me give you an example. For example, if you are running long-running workflows, right? So the mechanism by which all the relevant state is passed from an existing workflow to a new workflow with fresh event history to avoid unbounded growth of the event history, I think we did not have to rewrite any of this. All of this came sort of out of the box in Temporal. These were the primitives that we were just... able to use. Every workflow execution has a history stored in a database. This means that we have full immutable access of audit trail with an identity attached. So that made our risk teams very happy because we were able to provide access patterns and show them where things were. I wanted to share, since we are running Temporal since early 2024, we were the first team in capital one that pioneered the usage of durable executions the platform has grown since then now we have together with our partners we deliver 50 plus infrastructure capabilities we have 100 plus contributors on our platform and we have seen massive developer productivity gains the things that used to take two weeks to two months to even a year now can be done in days if not hours so that that was incredible this approach also sort of helped us deal with convey's law so usually what happens is teams are named after the products they build and then those teams reorg and when they reorg they don't support those products anymore now these products were part of this big platform and so now that you know you can write your code in a set of workflows and you just manage those workflows so we sort of obviated and have it demarcation point with the org reorgs that happens usually so this was really good platform ecosystem i know that self-hosting temporal worked for us will it work for you that's the question that you all have to decide but here are some guiding principles that helped us should you opt for a choreograph delivery solution or orchestration well it depends if it's a very simple thing that you have to do orchestration might be overkill. You might just go by running some lambdas and choreographed delivery would just be fine for you. How should you approach building, deploying, and managing? Because there is a lot of work, as Joseph mentioned, from auth, from getting things right. How do you manage some of these deployment patterns? Do you want to go self-host versus manage native service from Temporal? Or do you want to consider some other things that are out there? based on your use case does this fit your enterprise vision and strategy this is the big one because can you centralize the adoption of temporal as namespace as a service or you want individual flowers to bloom where you have teams do whatever they would like however they would like from an open source implementation point of view do you need different teams participating collaborating into a namespace boundary Think about Nexus. So those are some of the things that I think you all would have to think about if you were to go on the path of building Temporal and hosting it by yourself. Now, I'm going to leave this slide here for a little bit, and we can talk about a couple of items here. Temporal open source best practices from our implementation, these are our hard-won lessons. We found that developers, by default, don't think in the durable execution way of thinking. So you have to train. Developer training was essential. That's where Temporal sort of helped us. We had developer trainings for our team who are managing the platform. Temporal also provides certain primitives. But are you using those primitives the right way? For example, are you using workflows, activities, child workflows, et cetera, the right way, the way they are described? Do you want to call external APIs in the workflow or activities, things of that nature? doesn't come naturally because durable execution is a after-learned process. One more thing here to mention is that exposing certain capabilities as common workflows are important. It's definitely important because, for example, if you are building a Slack notification thing, but 10 other teams are also doing that, that's just causing more work. So what we have done in our platform is opening a production change. building your dashboard, things of that nature. We have sort of automated into a common namespace and gave common worker access to everybody so that they could utilize those things. They don't have to redo stuff again and again. And the last thing here on generic bullet is code review checklist. Using temporal activities, workflow, signal, update, heartbeat, and timeouts is just You have to come up with best practices so that the teams building these solutions can build at the same standardized way. And speaking of timeout, which timeout to use? Temporal have like schedule to close, start to close, and I don't even know what else. So I feel like you have to standardize some of these so that the temporal primitives can be standardized and used carefully throughout the platform. Now let's talk a couple of things about workflows. Workflows must be deterministic. What does that mean? This means that given the same request and the same steps will be repeated to produce the same result each time. We saw that when developers were building on temporal in the beginning, they were writing all the code in workflow, which was not the right thing to do because, you know, workflow doesn't, then you have to think about retries. If it crashes, then you lose everything. So most of your code should go in. activities. So those are the kind of things, like you must set your workflows with the right timeouts, et cetera. You could version your workflow. If you are running a short-lived workflow, you probably don't need to go versioning. You probably don't need heartbeats. But if you are running a long-running workflow, you probably need heartbeats. You probably need versioning. You probably also have to think about event history and start as new, continue as new workflow, et cetera. In the activity, so one thing to think about here is activities are for doing and workflows are for deciding. So everything, the heavy lifting of our code goes into the activity. All our business logic goes into the activity. Do not add retry logic into the activity. Temporal already retries the activities and have a default pattern like many other distributed systems with at least one guarantees built in the execution. So I know that we have thrown a lot at you, and I think we stand between you and the replay party. So I'm going to close this with just one slide. And if you take away one thing from this session, consider this. How many of you have heard Werner Vogels? And you know this quote, that everything fails all the time. Werner is the CTO of Amazon, and he always talks about this. But I want to extend this quote a little bit. Durable executions can help get you ahead of that. So with that, I want to really thank you all for coming here and listening to us. I hope this was helpful. I do want to thank a few folks from Temporal. Thank you, everyone, for your time. Ryan, Lambert, Rick Ross, Tiho, Alex, Carvo. and stew thank you everyone for partnering with us i think we really appreciate your your support thank you
⚙️ Pipeline jobs
| Stage | Status | Att. | Updated | Error |
|---|---|---|---|---|
| download | done | 1/3 | 2026-07-20 14:55:50 | |
| transcribe | done | 1/3 | 2026-07-20 14:56:23 | |
| summarize | done | 1/3 | 2026-07-20 14:57:00 | |
| embed | done | 1/3 | 2026-07-20 14:57:03 |
📄 Описание YouTube
Показать
Vivek Gupta (Director, Distinguished Engineer) and Joseph Gilbert (Sr. Manager, Engineering) from Capital One detail how the team modernized their cloud provisioning platform by moving from static infrastructure and choreographed deployments to a durable execution model on Temporal Open Source. They cover the architectural principles behind the shift, the reliability and observability gains, and the lessons learned running Temporal open source in a regulated enterprise environment. Filmed at Replay 2026: The Durable Execution Conference for AI, by Temporal. Temporal is the simple, scalable, open source way to write and run reliable cloud applications. Learn more Blog: https://temporal.io/blog How Temporal Works: https://temporal.io/how-temporal-works Community Slack: https://temporal.io/slack Developer resources Docs: https://docs.temporal.io Courses: https://learn.temporal.io/courses Support forum: https://community.temporal.io