Golang Microservices Full Course #33 - Outbox Pattern

Ikhda Muhammad Wildani · 2026-07-15 · 36м 22с · 24 просмотров · YouTube ↗

Топики: durable-execution

🎧 Аудио

📝 Summary

model=deepseek-v4-flash · prompt=summary-v7 · 5 603→2 963 tokens · 2026-07-20 15:03:34

🎯 Главная суть

Паттерн Outbox решает фундаментальную проблему распределённых транзакций в микросервисной архитектуре: как гарантировать, что запись в локальную базу данных и отправка события в брокер сообщений (RabbitMQ, Kafka) произойдут атомарно. Решение — записывать событие в отдельную таблицу outbox в той же транзакции БД, а фоновый Worker асинхронно публикует эти события в брокер, обновляя их статус. Это обеспечивает гарантированную доставку сообщений даже при временных сбоях брокера.

От монолита к микросервисам: цена выбора

В монолите одна БД и одна транзакция — ACID гарантирует атомарность: при ошибке rollback, при успехе commit. В микросервисах каждая служба имеет свою БД, поэтому распределённая транзакция невозможна. Приходится вводить саги (saga) с компенсацией, паттерн Outbox, circuit breaker, retry, DLQ. Сложность перемещается из кода в инфраструктуру и сетевое взаимодействие.

Сравнение подходов: монолит vs микросервисы

Аспект	Монолит	Микросервисы
Вызов между компонентами	Функция в том же процессе (микросекунды)	gRPC/HTTP (миллисекунды, возможны сбои)
Деплой	Единый бинарник, при рестарте останавливается всё	Каждый сервис деплоится независимо
CI/CD	Простой	Сложный, требует оркестрации
Отладка	Call stack — легко трассировать	Нужен distributed tracing (trace ID, correlation ID) — log aggregation (ELK), заголовок X-Request-ID
Частичный отказ	Либо всё работает, либо всё упало	Сервис A работает, B упал, C таймаут — логика ломается

Когда микросервисы оправданы

Крупная команда (5 команд по 4–20 разработчиков, каждая ведёт свой сервис).
Масштабирование по фичам: например, wallet service требует 100 инстансов, а ledger — только 5.
Изоляция: сбой wallet не валит auth service.
Разные технологические стеки: wallet на Go, notification на Python, auth на JavaScript.

Когда микросервисы не нужны

Маленькая команда (1–5 разработчиков).
Низкий трафик — нет потребности масштабировать сервисы отдельно.
Команда не знакома с распределёнными системами.

Рекомендация: начинать с modular monolith — модули разделены логически, но деплоятся как один бинарник. При необходимости модуль можно вытащить в отдельный микросервис.

Конкретный пример: перевод денег

В монолите операция перевода (wallet debit → wallet credit → ledger record) выполняется в одной транзакции. В микросервисах те же шаги требуют саги, Outbox, circuit breaker, компенсаций.

Проблема атомарности при отправке уведомления

Транзакция перевода успешно завершается, статус в БД — success. После commit нужно опубликовать событие transfer_completed в RabbitMQ, чтобы Notification Service отправил email пользователю. Возможны два плохих сценария:

DB commit успешен, publish упал (RabbitMQ недоступен) — деньги списаны, email не отправлен. Пользователь жалуется.
publish успешен, DB commit упал (ошибка БД) — email отправлен (пользователь видит «успешно»), но транзакция откатилась. Деньги не переместились, пользователь в замешательстве.

Сетевая операция (publish) не может быть включена в транзакцию БД — это и есть «разрыв» (gap), который закрывает Outbox.

Как работает Outbox — гарантированная доставка

Шаги в одном запросе:

Начать локальную транзакцию БД.
Обновить статус в основной таблице (например, transactions → success).
Сохранить событие в таблицу outbox_events (event_id, type, payload, status='pending', attempts=0).
Завершить транзакцию — атомарно фиксируются оба изменения. Если что-то пошло не так — rollback всего.
Фоновый Worker (вне потока запроса) периодически читает из outbox_events записи со статусом pending.
Публикует событие в RabbitMQ.
При успехе — обновляет статус на sent (или удаляет запись).
При ошибке — увеличивает счётчик попыток и повторяет позже. Если брокер недоступен 10 минут — события остаются в БД, не теряются.

Реализация на Golang (из практической части)

Миграция БД: создание таблицы outbox_events:

CREATE TABLE IF NOT EXISTS outbox_events (
    id SERIAL PRIMARY KEY,
    event_type VARCHAR(255),
    payload JSONB,
    status VARCHAR(50) DEFAULT 'pending',
    attempts INT DEFAULT 0
);

Модель (internal/transaction/model/outbox.go):

EventID
EventType
Payload
Status
Attempts

Репозиторий — добавляются методы:

CreateOutboxTx(tx, event) — вставка в рамках транзакции.
UpdateStatusTx(tx, eventID, status) — обновление после отправки.
ReadOutboxPending(ctx) — выборка записей со статусом pending.
GetOutboxEventsWaiting(ctx) — для worker.

Сервисный слой: в функции перевода (Transfer), после успешного debit/credit/ledger записи, до commit вызывается CreateOutboxTx, который сохраняет событие transfer_completed. Commit — атомарно. Если commit прошёл — событие точно сохранено.

Worker (не реализован в этом видео, только упомянут): будет читать outbox_events, публиковать в RabbitMQ и обновлять статус.

Тестирование в Postman

Авторизация (логин в wallet).
Перевод 500 единиц получателю (transaction ID).
Проверка таблицы outbox_events — запись со статусом pending, 0 попыток, тип transfer_completed.
Транзакция в transactions — статус success.
Баланс кошелька отправителя уменьшился, получателя — увеличился.
Ledger mutation — reconcile: true.

Результат: данные консистентны, событие ждёт отправки. Далее нужно реализовать Outbox Consumer, который будет публиковать события в RabbitMQ и обрабатывать повторные попытки.

Дополнительные концепции, затронутые в видео

Distributed Tracing: для отслеживания запроса через несколько сервисов используется trace ID (correlation ID) — уникальный идентификатор, внедряемый в заголовки (например, X-Request-ID). Связка с ELK (Elasticsearch, Logstash, Kibana) позволяет искать логи по этому ID.

Retry & Circuit Breaker: необходимы при сетевых вызовах между сервисами, чтобы обрабатывать временные ошибки.

Saga Compensation: если шаг в саге упал после успешного выполнения предыдущих шагов, нужно выполнить компенсирующие действия (например, вернуть деньги обратно). В данной реализации Outbox используется совместно с сагой.

DLQ (Dead Letter Queue): если после всех попыток отправить событие не удаётся, оно помещается в очередь недоставленных сообщений для ручного анализа.

📜 Transcript

en · 3 391 слов · 67 сегментов · clean

Показать текст транскрипта

hello back again to this kolang microservices full course and in this video i want to share about outbox pattern and before we talk about outbox patterns i want to share to you about what is the problem inside monolith and then microservices so first we have data consistency which is in the monolith is easy because one db one transaction will be acid atomic acid is atomicity acid is no acid in software is atomicity and consistency isolation and durability if we are have working on the monolith this is very easy because we only have one db and we can commit or roll back if anything wrong we can roll back if anything if anything works we can commit but in the micro services it's very hard because we have separate db we need a saga without which is the last time that we have already talked about saga compensation and yeah saga pattern and saga compensation so and then we need an outbox pattern which is i we want to implement the we want to learn and implement in this video and then after that the second is network call so for the monolith function call is microseconds and then for the microservices jrpc or http which is millisecond but can be down sometimes yeah one of them one does one of the service can be down and it's will breaking right that's why we need to saiga and then we need an outbox and therefore the deployment we only one binary and then we if we restart we restart all of the deployment in the monolith but for microservices can be deployed per service and but for the CI CD is very complex to set up and then after that communication and communication for each service is for the monolith it's only execution code in the same process such as wallet debit wallet credit and ledger record yeah something like that but in the microservice it's need a gRPC or RedquidMQ, need a circuit breaker, need a retry mechanism, need a DL key and this is very hard if we are new to the software development and then after that for the debugging we can want call stack so it's easy to trace but for the microservice because the request is spread across the services so we need a distributed tracing what is distributed tracing distributed tracing is this three booted tracing is observability technique to monitoring and tracking the request because it will travel around across the services database and network right in the microservice architecture so we need this So we need something like trash ID to know yeah correlation ID also can use for this if we search what is used about correlation ID in the distributed tracing so it's synonym as a trash ID and this is a unique identifier that this is from the start of the request and when they are across the services this still need to be forwarded yeah so we can trace we can see the log for that ID trace ID or correlation ID this is good if we have ELK stuck for the logging and monitoring So the logging tracing will be much easier because we know what is the ID that we have correlation ID right and for this one is Injected in the X request ID Maybe and the benefit of using is debugging and end-to-end feasibility and system context So yeah, then after that how to set up the dev for the model we just go run main.go but for the microservice we need to go run main.go all of the service if we have five service then we up five service so we use docker-compose to make it easier to up to go run and then instead of instead of open the five terminal to run the main.go yeah and in every surface then partial failure is in the monolith either is all down or all up so if the system down the monolith will cannot accept anything but sorry but in the micro surfaces if surface A up B down it can be happen or C timeout it also can be happen so the logic will be crash and this is the threat of if you are move the complexity from code to the network or infra infrastructure so either you choose complex in the code or you just complex in the infrastructure it's up to you but in the monolith the big Tangler complexity is in the code but in the microservices is complexity in the network communication consistency that is the problem that we will face in the microservices the microservices project yeah so if you see here in the monoliths if we are db while service db we only call the function like wall service db So this will be one DB transaction but in the microservice update wallet balance DB can be wallet client update wallet balance and then we need to add circuit breaker, we need to add retry, we need to add compensation for Saga we need to add outbox pattern because no distributed transaction and we need to DLK if all of the function is broken or failed so the old patterns we will implement in this episode isn't because this is not a feature this is the price that we need to pay because choosing microservices in the monolith we don't have this problem but when microservices have worth it to try first if you want to have a microservice That's what it to do if you have a big team like a five team 20 plus dev so every every team deploy every service deploy each their service so the and so like one team is 20 dev or one team is four dev so we have a five different team one team Working on the different service. It's what it to work with the microservice and then after that if it's not worth if we have a small team like one or five dev and then after that the pros is if worth it for the microservices you will want to scale per feature like a wallet service need a hundred instance but ledger only need a five instance and need an isolation if service wallet down the out service will not go down as well and then text tag is different like service one is golang service two is notific service two is python such as like maybe wallet is golang and then python is for out service or something and then python is for out service or notification service and for the javascript can be notification service and yeah if you want a text stack different different text stack you can use some microservice and that's not worth use microservices you have a small team and then the traffic is still low and then you don't need a scalper service and the team is not familiar with distributed system so yeah the rules is first monolith and then we can extract to service only if we need that so we think the best thing if you want to start with monolith i can recommend you using monolith modular which is the module is padded but only one deploy so if any have anything in the future we want to scale we can extract the module to the service and this is what you want to try right so I think here I want to share about the monolith function first is transfer which is wallet debit credit and record for the ledger so this is all in the monolith we will do with a database transaction so first is commit sorry first is start then after that is commit or rollback right and then after that in the microservice we need to update the balance we need to saga we need to yeah that's what we are doing last time right try compensation output pattern and circuit breaker and then the L key right so this is we talk about right now and I think I haven't I have done for the theory and like I said I want to work on the outbox patterns for distributed transaction so outbox pattern is a solution for the distributed transaction in microservices architecture so the core is to ensure the data is consistent without a without losing any message here this is guaranteed message delivery so if service a service updating the database and want to publish event to the message brokers such as rapid mq or kafka these two operation is not guaranteed to be atomic because this is um asking a different system like a db and network broker and this outbox patterns outbox pattern is want to fill the gap. So we want to write an event to the local table in the database. So the other worker will send the event from the table to the message broker. So when we need an outbox pattern, so if the transcription service is processing the transfer, and status updated to success, we want to publish the event transfer completed to the RabbitMQ. So the notification service can send to the email asynchronously. What is the problem? What happened if the publish to RabbitMQ is failed after the database transaction commit and commit the status to the success? Because the money was deducted but the email has never sent So this is the bad scenario if this happen first DB commit is success but publish fail which is update transaction set status is success but publish RebitMQ is fail because the broken down money is deducted but not if email is never sent user will complain and then the scenario 2 publish success but DB fail update transaction success this is the fail one db error maybe and publish rebitmq is success so email has sent transfer successful but transaction fails so user will confuse why my money why my transaction is success but my money never come in this network call cannot be including in the database transaction so this is the gap So what how how the outlock patterns work? So first DB local transaction is begin and then save update transaction status in the table transaction and then save event in the table outlock events So after that DB local transaction will be commit hundred percent atomic This one is because one DB connection if one transaction is failed the both will be rollback. So after that the worker the background process not in the request flow will get from the outbox events and then publish to RabbitMQ and then update the status and send the email maybe yeah and then if the broker down for a 10-minute event still in the outbox events so this is the this is current delivery because even the broker is down the data still in the database and can be doing again redo again for the transaction transaction notification sorry okay so I think we can start first I want to create new database um database migration here i think i want to create try to migrate migrate one this one okay okay this one i cd to the microservice and wallet sorry transaction service right yeah in the transaction service i do here migrate the create outbox table so here we get a new file inside this one I want to copy drop table is if X is and then I add the outbox table if don't X is here and then I create the model inside the internal transaction model what is this outbox.go outbox event ID event type payload status and attempts then after that inside the repository at another function we did which is create the x update status tx under here and then create outbox tx here so we need to add the function here after create yeah I think here the difference is this is tx sql tx but this is rdb context so that is the difference and I want to add this one Where I can add here should be fine. Yep, and it should be good now the surface dot go I need to go to the line hundred and ten Service dot go service test first Service test first. I need to add sql mock here because we had an error here sql mocks and this new DB test here and after 18 yeah 18 is here okay SQL DB and then third that outbox event I add inside the here moxdx repo okay delete then the create the X create the X here and update status DX update status DX here then read outbox DX and get outbox events waiting very stiff hundred here okay and also here in certain new transaction service this is nail I need to create this one Change to this and then outbox inside 221 221 221 221 here I think after save here safety x nil at this one So outbox a single pending transfer complete event must be record Okay, then in the transaction service again inside 250 here spending only and then no outbox this should be after tx yeah this one and after the this one 3313 313 here and set 379 379 is where here but we have an error because we should add this one where is my terminal go get this one data doc okay should be no error now yeah good then after that okay so the service I think I add again here 110 where is 110 110 is here I think create record we do this in space short transition to release database log quickly inside here we here is dx repo grid yeah dx repo grid delete yeah this error is inside 121 so yeah okay only until error okay turn nil and execute this one is error wait I need to defer all back here until 5 wait is 5 this is 1 2 3 this is wait this is 1 4 9 should be here I think yeah wait what happened is this transaction to put function transaction is here what happened with this one let me check Let me check this is inside itempotency key and txrepo create this one right transfer we do this in spirit we add versus init until here I wanted to copy where is this we change the txrepo create here to this and execute grpc transfer chain is not is not here but we have the function after that we add the different back it should be no error but this is different give it amount I think like amount.nec I think here like amount.nec I see okay so this is new function here okay this is the new function I want to copy this one first the X record here as this one and this is the new function inside here okay like this okay and we have an error which is this one the duck center balance wallet to change to the breaker okay I delete this one like this and then and then inside here return no custom error secret breaker is open here is copy status unavailable this one also copy here file transaction this one receiver user ID okay and then inside the x repo update status is remove remove yeah this one remove whether i need to check what is the remove one this one remove failed okay this one also remove inside this failed one okay this one should be removed and update status the X update status compensation felt also remove here set inside two three one yeah and then I copy here to here then amounts to string now we only have a month yep and after that what is the yeah delete the 4 5 2 4 this one 4 4 we compensation fail that is delete right compensation failed and then we return compensation for manual compensation here required intervention required and the x repo also we need to remove here also return we change the return wallet is available and then this one also and then still could receive user ID and amount and this one as well check again DX repost update status DX update status inside here also remove I think compensation local critical This one should be removed and I still have status internal server error I think we should change yeah, I think this is Receiver user ID I think yep, I want to follow first here No, I think we do it really is just deleting the middle one I think yeah one but we also delete the repo here sorry the tx repo and here amounts is also need to be changed then we have what receiver user id and here amount what we have again tx repo compensation update status if we see the x repo tx set executed grpc transfer chain yeah we have this one this is from transfer no no problem this transfer this one i just delete the name then month and then delete and then month then here then remove and then remove should check okay month here is id then here once a month we have to do it again remove remove remove and the DX record status finally here we need to what happened here we need to return nil okay we need to return but we have some error again I think we still have an error wait where is the error receiver user ID receiver user ID do we have an error here where an error I think no first is the x repo inside the top up inside top up inside transfer and set transfer okay I think this should be good we want to open the terminal what is the problem Should be gomot ID we do the make ID first we do the make CD first make ID and Stop the container we do the docker-compose This is build Wait, sorry make build Linux and Docker compost up this is build then the past the video. Okay, I already started I want to migrate so I do here which is transaction service and here then go to transaction I want to try up okay create a box pattern here and we can try we can test wait a second I want to verify inside our database which is inside call transactions I think outbox event yeah already here I want to open the postman outbox pattern when we login I think first we need to login okay login my wallet let me check okay my wallet here okay i think i want to send 500 500 500 500 sorry transaction transfer to firm again and seven yeah i think okay 15 lah to make it different okay here is the transaction let me check in the outbox pattern status is pending and we don't have an attempt yet and transfer completed that should be good test transaction 7 is success yeah okay this one is working fine and I want to check the wallet again of current user okay this should be good and I want to check the ledger mutation mutation yeah reconcile true still true transaction history data meta okay data dof last transaction is test transfer seven seven zero zero seven and then success okay i think this all for current video and for uh this video i hope you want you enjoy and think you can try to search outbox pattern in the Google what is that and what's that mean yeah something like that outbox pattern in Google is what let me check reliable messaging is this so outbox table yeah we have outbox table this is So handler using some transaction ACID using entity and outbox and then processor will do so I think This is the first thing that we are doing right saving order service to order table and outbox table we have a Transaction service which is to the transaction table and then outbox table yeah Then later we can try to make the outbox customer Okay, I think I hope we can do that then thank you see you in the next video

⚙️ Pipeline jobs

Stage	Status	Att.	Updated
download	done	1/3	2026-07-20 15:02:39
transcribe	done	1/3	2026-07-20 15:03:03
summarize	done	1/3	2026-07-20 15:03:34
embed	done	1/3	2026-07-20 15:03:35

📄 Описание YouTube

Показать

# Build a Wallet App with Go – EP33: Transactional Outbox Pattern

PR Github = https://github.com/bashocode/gowallet/pull/44

In Episode 33, we implement the **Transactional Outbox Pattern** to solve one of the biggest challenges in distributed systems: **the dual-write problem**. Instead of writing to the database and publishing an event separately, we'll store both the business data and the outgoing event in the **same database transaction**, ensuring reliable event delivery without relying on distributed transactions (2PC). This pattern is widely used in event-driven microservices to guarantee consistency between local database updates and asynchronous messaging.

## 📚 What You'll Learn

* What the Transactional Outbox Pattern is
* Understanding the dual-write problem
* Why distributed transactions (2PC) are often avoided
* Designing an Outbox table
* Writing business data and events in a single database transaction
* Publishing events asynchronously to a message broker
* Handling retries and failed message delivery
* Building reliable event-driven microservices in Go

By the end of this episode, your Wallet application will reliably publish domain events without risking data inconsistency, making your microservices more resilient, scalable, and production-ready. You'll also understand how the Outbox Pattern forms the foundation for event-driven architectures using brokers such as RabbitMQ or Kafka.

If you found this tutorial helpful, don't forget to:
👍 Like the video
💬 Leave a comment with your questions or feedback
🔔 Subscribe to follow the complete **Golang Microservices Full Course** and build a production-ready Wallet App from scratch.

#golang #go #outboxpattern #transactionaloutbox #microservices #eventdriven #rabbitmq #kafka #backend #walletapp