Event-driven architecture

Think of the notice board in an apartment building. When a package arrives at the front desk, the doorman does not go knocking on every door. He pins a note to the board, and whoever needs it comes by to read it: the resident of apartment 302 to pick up the box, the building manager to check the water bill, the caretaker to see whether the elevator is back. The doorman does not know who will read the note and does not expect an answer from anyone. If the caretaker is on vacation, the note stays up until he returns.

This is the way of thinking behind event-driven architecture, or EDA. A component records a fact that has already happened, hands that fact to an intermediary and keeps working. Anyone interested reacts later, at their own pace. The publisher does not need to know who consumes the event, and each consumer decides for itself what to do with the fact.

This lesson starts from the problem this architecture solves: direct calls between services. It then defines the vocabulary (event, command, broker), compares queues and logs, and goes into what really goes wrong: a message delivered twice, a message that arrives out of order, a database that saves the order while the broker never receives it. The solution patterns (idempotency, outbox, saga and event sourcing) appear as answers to these problems, without turning into a list of tools. The Go examples use only the standard library. The database and the broker are simulated in memory so that the logic stays visible, and the snippets were run with Go 1.27.2 on November 2, 2026.

Lesson map

1. Direct calls and their costs.
2. Command and event.
3. The broker.
4. Queues and logs.
5. At-least-once delivery and idempotency.
6. Ordering.
7. Dual write and outbox.
8. Saga.
9. Event sourcing overview.
10. Concerns when scaling.
11. Common mistakes and good practices.
12. When not to use it.
13. Exercises and glossary.

1. Direct calls and their costs

Most systems start with direct calls. One service needs something from another, makes an HTTP request or a gRPC call, waits for the response and moves on. This model is synchronous, and it has a clear advantage: the code says what happens and in what order. The example below models the creation of an order with three dependencies. It charges the card, records the loyalty points and sends the confirmation email, in that order.

package main

import (
	"errors"
	"fmt"
	"time"
)

func cobrarCartao(pedido string) error {
	time.Sleep(120 * time.Millisecond)
	fmt.Println("cartão cobrado:", pedido)
	return nil
}

func registrarPontos(pedido string) error {
	time.Sleep(30 * time.Millisecond)
	fmt.Println("pontos registrados:", pedido)
	return nil
}

func enviarEmail(pedido string) error {
	time.Sleep(50 * time.Millisecond)
	return errors.New("serviço de e-mail indisponível")
}

func criarPedido(id string) error {
	if err := cobrarCartao(id); err != nil {
		return err
	}
	if err := registrarPontos(id); err != nil {
		return err
	}
	if err := enviarEmail(id); err != nil {
		return fmt.Errorf("pedido %s: %w", id, err)
	}
	return nil
}

func main() {
	err := criarPedido("42")
	fmt.Println("resultado:", err)
}

The output of one run, with the email service down, is this:

cartão cobrado: 42
pontos registrados: 42
resultado: pedido 42: serviço de e-mail indisponível

The charge went through and the points were recorded, but the order ends with an error. The customer sees a failure, tries again and is charged a second time. The three parts did what they were supposed to do. The defect lies in the sale depending on an email, a task that should not decide whether the purchase happened. A design like this is often called an architecture, but it looks more like a bank queue in which each teller handles everything before serving the next person.

Coupling is the degree to which one component depends on another to work. The order case is temporal coupling: the three services must all be available at the same time for the sale to finish. The response time of the order is also the sum of the time of all of them. Each new step increases that sum and increases the number of places where the sale can break.

How the event cuts the dependency

In an event-driven architecture, the order service stops calling the email service. It records that the order was created, publishes that fact and answers the customer. Whoever cares about the fact subscribes and reacts. If the email service is down, the sale is already complete and the confirmation waits in the broker until the service comes back. The two figures below show the chained design and the event design.

In figure 01, the order fires three calls in sequence, each one after the previous one: charge, points and email. The email arrow appears blocked because the service is down. The whole sale stops at that arrow, even though the card has already been charged.

In figure 02, the order publishes the PedidoCriado fact to the pedidos topic and moves on. Stock, points and email consume the same event through separate paths, each with its own consumer group. The email shows a growing delay, but the order does not notice, because it finished before any consumer started.

2. Command and event

There are two kinds of message that are easy to confuse. A command is an order given to someone: “charge this card”, “send this email”. It has a definite recipient and expects that recipient to do something. An event is a fact that has already happened, and its name is in the past tense: PedidoCriado, PagamentoRecusado, EstoqueReservado. Whoever publishes an event does not tell anyone to act. It reports what happened, and whoever is interested decides what to do with the information.

The difference looks like a matter of vocabulary, but it changes the coupling. When the order service sends the EnviarEmail command, it needs to know that an email service exists, what its name is and what it expects to receive. When it publishes PedidoCriado, it only needs to know that the order was created. If the marketing team wants to listen to the same event to send a discount coupon, the order service does not change a single line. Renaming EnviarEmail to PedidoCriado, however, does not turn the code into an event. It still tells someone to do something, just in a quieter voice.

The table summarizes the differences that weigh most in day-to-day work:

The last row of the table leads to a rule that sounds bureaucratic and is essential: a published event is never edited. If the charge was refunded, CobrancaEstornada is published, and PagamentoAprovado stays in the history as a fact that happened. This rule comes back in section 9, when the history becomes the source of truth of the system.

A good event carries enough for the consumer to decide without calling the producer back: a unique identifier, the type, the moment it happened, the version of the contract and the minimum data. The size of the payload is a choice with a cost, and section 10 returns to this.

3. The broker

The intermediary that receives the events and delivers them to those interested is called the broker. It solves two decoupling problems. The first is in time: the producer can publish even when the consumer is offline, and the consumer reads when it comes back. The second is in space: producer and consumer do not need to know each other’s address, language or number.

Brokers differ in how they deliver. A queue hands each message to one consumer among several that compete for the work, as the queues of RabbitMQ and Amazon SQS do. A publish/subscribe topic, described in publish/subscribe, copies the message to all subscribers. A log, such as Kafka, stores the messages in sequence and lets each consumer group read at its own pace. Section 4 compares queues and logs in more detail. Many products mix these modes, and the choice depends on what should happen to the message after delivery.

The broker becomes the place where the events are kept. That makes its availability and durability central subjects, because losing the broker means losing facts that the whole system already treated as recorded.

4. Queues and logs

A queue works like the line at a supermarket checkout. Each customer is served by one cashier, and when the service ends, the customer leaves the line. In messaging, the consumer takes a message, processes it and confirms with an ack, and only then does the broker remove it. If several consumers are attached to the same queue, they compete for the messages, and each message is processed by only one of them. This is the ideal model for distributing work: converting a video, issuing an invoice, resizing an image.

A log works like a shop’s ledger. Entries are written in order and numbered, and nobody erases a line because an auditor has read it. Each auditor keeps only the page where they stopped, which in messaging is called an offset. Because the record stays available for a configured period (by time or by size), a new consumer can start from the beginning and reread the past. The auditor who asks for the book from the start does not get in the way of the day’s cashier, and this is the central gain of the log model.

Figure 03 places the two models side by side. On the left, the queue hands tasks to two workers and, after the ack, the task disappears. On the right, the pedidos topic keeps the messages, and each consumer group has its own offset: finance is at point 3, and analytics, which started later, rereads from the beginning.

The figure shows the difference that matters most: in a queue, reading means consuming, and everything confirmed disappears. In a log, reading deletes nothing, and a new group can start from zero. The table compares the aspects that usually decide the choice:

No model is better in general. A queue is simpler when the job is distributing tasks and the message has no use after it is processed. A log pays off when the fact interests several systems, when the history matters and when someone will need to reprocess everything months from now.

5. At-least-once delivery and idempotency

When a message leaves the producer, there are three possible delivery promises. At most once means the message may be lost, but it is never processed twice. At least once means the message is never lost, but it may be processed more than once. Exactly once means the effect happens a single time, which is harder to guarantee than it seems.

The reason at-least-once is the most common choice is simple. The broker does not know whether the consumer processed the message. It only knows whether it received an ack, and the ack can be lost on the network. It can also happen that the consumer processes the message, writes to the database and crashes before confirming. In both cases, the broker, which cannot leave a message unconfirmed, delivers it again. Figure 04 shows this sequence, with the ack lost on the way.

The timeline reads from top to bottom. The consumer applies the effect of pgto-7 and tries to confirm, but the ack is lost (the cross on the arrow). The broker, getting no confirmation, delivers the same event again. Without any protection, the effect would be applied twice. With protection, the consumer notices the repetition and simply confirms.

The protection is called idempotency. An operation is idempotent when repeating it produces the same effect as running it once. One rule for this is to store the event identifier together with its effect, and to check that record before applying anything. The example below makes this comparison: a consumer without protection next to another one with a table of processed events.

package main

import (
	"fmt"
	"sync"
)

type Evento struct {
	ID    string
	Tipo  string
	Valor int
}

// Consumidor sem deduplicação: aplica o efeito a cada entrega.
type ContaSemDedup struct {
	saldo int
}

func (c *ContaSemDedup) Handle(e Evento) {
	c.saldo += e.Valor
}

// Consumidor idempotente: guarda o ID do que já processou.
// Em um banco real, o saldo e a tabela de processados entram na mesma transação.
type ContaComDedup struct {
	mu          sync.Mutex
	saldo       int
	processados map[string]bool
}

func NovaContaComDedup() *ContaComDedup {
	return &ContaComDedup{processados: make(map[string]bool)}
}

func (c *ContaComDedup) Handle(e Evento) {
	c.mu.Lock()
	defer c.mu.Unlock()
	if c.processados[e.ID] {
		fmt.Printf("  %s já processado, ignorado\n", e.ID)
		return
	}
	c.saldo += e.Valor
	c.processados[e.ID] = true
}

func main() {
	pagamento := Evento{ID: "pgto-7", Tipo: "PagamentoAprovado", Valor: 100}

	// O broker entrega duas vezes: o ack da primeira entrega se perdeu no caminho.
	entregas := []Evento{pagamento, pagamento}

	sem := &ContaSemDedup{}
	for _, e := range entregas {
		sem.Handle(e)
	}
	fmt.Println("sem deduplicação, saldo:", sem.saldo)

	com := NovaContaComDedup()
	for _, e := range entregas {
		com.Handle(e)
	}
	fmt.Println("com deduplicação, saldo:", com.saldo)
}

The output shows the difference:

sem deduplicação, saldo: 200
  pgto-7 já processado, ignorado
com deduplicação, saldo: 100

Figure 05 describes the decision the idempotent consumer makes for each event that arrives.

The diamond asks whether the event ID is already recorded. If it is, the consumer confirms and ignores the event. If it is not, the consumer applies the effect and records the ID in the same transaction, and only then confirms. This last requirement is the part that most often fails. If the balance is updated in one database and the ID is recorded in another, a crash between the two writes leaves the system in one of two wrong states: the effect applied without a record, which causes duplication on the next delivery, or the record without the effect, which loses the event.

Some tools offer their own guarantees, and it is worth knowing exactly where they stop. Kafka’s idempotent producer, enabled by default since version 3.0, prevents the producer’s own retries from writing the same message twice to the log. It does not know your balance or your email, so it does not prevent the repetition that happens on the consumer side. Amazon SQS deduplication in FIFO queues works with a 5-minute window: a resend with the same deduplication ID, made within that window, does not enter the queue as a duplicate. AWS’s own documentation describes the sending side and does not address the effect the consumer produces afterwards. End-to-end exactly once, including the database and the email, is almost always built by whoever writes the consumer.

6. Ordering

Messages about the same entity can arrive out of order. A PedidoCancelado may be processed before PedidoCriado if the first was published by another server, or if the created event suffered a new attempt after an error. Parallelism also shuffles the order: two consumers reading the same queue concurrently do not guarantee that the second message finishes after the first.

Guaranteeing global order, that is, a single sequence for all the events in the system, is expensive. It requires a serialization point, and that point limits scale. That is why order is usually guaranteed in smaller scopes. In Kafka, order holds within each partition, and the message key decides which partition the message falls into. Messages with the same key land in the same partition as long as the number of partitions does not change, and the order of that entity is preserved. Figure 06 shows this with two orders.

The figure shows order 42 with created and paid in the same partition, P2, while order 77 falls into P1. The consumer reads P2 in the order the messages arrived, so it sees created before paid. There is no order guarantee between P1 and P2, and that does not matter for order 42, because it depends only on itself.

When the transport cannot guarantee order, there are three common ways out. The first is to choose a partition key that matches the entity. The second is to number the changes with a version and refuse any event that is not the next expected one, as the append function in the section 9 example does. The third is to design the consumer to tolerate the wrong order. Recording that the customer changed address tolerates disorder well, because the last information prevails. Adding ten loyalty points tolerates none of this, because the sum depends on the order in which the installments arrive, and an unprotected repetition adds them again.

7. Dual write and outbox

A service that creates an order has to do two things: write the order to the database and publish the PedidoCriado event to the broker. These are two different systems, and no transaction covers both without heavier protocols. This is the dual write problem. Figure 07 shows the risk window: if the service writes and crashes before publishing, the order exists and the event never leaves.

The figure shows the most common case. The order commit happens, and right after it the publication fails. The result is a real order with no event, and stock, points and email never learn about it. If the order were reversed, publishing before the commit, the risk window would switch sides: the event would go out for an order that the transaction later undid. Both paths leave the system inconsistent, and no error-handling code solves this alone, because the crash can happen between any two lines.

The most used solution is the transactional outbox pattern. Instead of publishing directly, the service writes the event to an output table (the outbox) inside the same transaction that writes the order. Either both rows are written, or neither is. A separate process, the relay, reads the pending outbox rows, publishes them to the broker and marks each one as published. If the relay crashes in the middle, the row stays pending and is published on the next run. This is the mechanism that Chris Richardson’s site describes as an answer to the atomicity problem between database and broker, without resorting to two-phase commit. The relay can be a periodic poller of the table (polling publisher) or a reader of the database’s transaction log (transaction log tailing), and both forms appear in the literature.

The cost of the outbox is that delivery becomes at-least-once. If the relay publishes and crashes before marking the row, it publishes again. The example below simulates exactly this crash: the first run of the relay publishes and crashes, and the second publishes the same row once more.

package main

import "fmt"

type Evento struct {
	ID   string
	Tipo string
}

type LinhaOutbox struct {
	Evento    Evento
	Publicado bool
}

// Banco simulado: o pedido e a linha da outbox são gravados juntos.
type Banco struct {
	pedidos []string
	outbox  []*LinhaOutbox
}

func (b *Banco) CriarPedido(id string) {
	// Em um banco real, as duas inserções ficam dentro de uma mesma transação.
	b.pedidos = append(b.pedidos, id)
	b.outbox = append(b.outbox, &LinhaOutbox{
		Evento: Evento{ID: "ev-" + id, Tipo: "PedidoCriado"},
	})
}

type Broker struct {
	recebidos []Evento
}

func (br *Broker) Publicar(e Evento) {
	br.recebidos = append(br.recebidos, e)
}

// O relay publica as linhas pendentes e só depois marca cada uma como publicada.
// Se ele cai entre a publicação e a marcação, a linha continua pendente.
func (b *Banco) Relay(br *Broker, cairAntesDeMarcar bool) {
	for _, linha := range b.outbox {
		if linha.Publicado {
			continue
		}
		br.Publicar(linha.Evento)
		if cairAntesDeMarcar {
			fmt.Println("relay caiu depois de publicar", linha.Evento.ID)
			return
		}
		linha.Publicado = true
	}
}

func main() {
	banco := &Banco{}
	broker := &Broker{}

	banco.CriarPedido("42")

	banco.Relay(broker, true)
	banco.Relay(broker, false)

	fmt.Println("pedidos gravados:", banco.pedidos)
	fmt.Println("eventos recebidos pelo broker:", len(broker.recebidos))
	for _, e := range broker.recebidos {
		fmt.Println(" -", e.ID, e.Tipo)
	}
}

The output:

relay caiu depois de publicar ev-42
pedidos gravados: [42]
eventos recebidos pelo broker: 2
 - ev-42 PedidoCriado
 - ev-42 PedidoCriado

The event ended up twice in the broker, and the consumer from section 5 must absorb the repetition. Figure 08 shows the complete flow, from the order to the consumer.

In figure 08, the local transaction writes the order and the outbox row inside the same group. The relay reads the pending rows, publishes them to the topic and marks each one. The consumer, on the right, receives the delivery and must accept repetition. The arrow to publishing goes back and forth to show the broker’s confirmation, which must exist before the relay marks the row as sent.

Anyone who says they never forgot to publish an event after the commit has not yet spent a night on call. With the outbox, forgetting no longer depends on someone remembering: the outbox row is already written together with the order.

8. Saga

A transaction that spans several services, each with its own database, has no global COMMIT at hand. Protocols such as two-phase commit exist, but they require a coordination that many databases and brokers do not offer, and they lock resources while they wait. The saga was proposed by Garcia-Molina and Salem in 1987, in the paper “Sagas”, for long transactions, and today it is the most used answer to this case in service architecture. The idea is to split the transaction into local steps, each with a compensation: the action that undoes the step if something goes wrong after it.

When a step fails, the saga runs the compensations of the steps already completed, in reverse order. If the stock was reserved, the charge was made and the invoice failed, the saga refunds the charge and then releases the stock. Figure 09 shows the execution with step 3 failing.

The figure shows step 3, issuing the invoice, marked as failed. Steps 1 and 2 have compensations (undo), shown on the right, and they run in reverse order: first refund the charge, then release the stock. Step 4, the email, never runs, because the saga stopped before it. The compensation of step 3, cancelling the invoice, does not run either, because the failed step never completed.

There are two ways to coordinate a saga. In choreography, there is no coordinator: each service listens for the event of the previous one, runs its step and publishes its own event. It is decentralized, but the full flow ends up scattered and hard to follow as it grows. In orchestration, a central component sends commands to each service, receives the responses and decides the next step. It is easier to follow, and the orchestrator becomes a component that needs extra care. The example below implements orchestration in its simplest form.

package main

import (
	"errors"
	"fmt"
)

type Passo struct {
	Nome    string
	Executa func() error
	Desfaz  func() error
}

func executarSaga(passos []Passo) error {
	var feitos []Passo
	for _, p := range passos {
		fmt.Println("executando:", p.Nome)
		if err := p.Executa(); err != nil {
			fmt.Printf("falhou em %s: %v\n", p.Nome, err)
			for i := len(feitos) - 1; i >= 0; i-- {
				fmt.Println("compensando:", feitos[i].Nome)
				if err := feitos[i].Desfaz(); err != nil {
					fmt.Println("compensação falhou, precisa de nova tentativa:", err)
				}
			}
			return err
		}
		feitos = append(feitos, p)
	}
	return nil
}

func falha(msg string) func() error {
	return func() error { return errors.New(msg) }
}

func ok() error { return nil }

func main() {
	passos := []Passo{
		{"reservar estoque", ok, func() error { fmt.Println("  estoque liberado"); return nil }},
		{"cobrar cartão", ok, func() error { fmt.Println("  cobrança estornada"); return nil }},
		{"emitir nota fiscal", falha("serviço fiscal fora do ar"), func() error { return nil }},
		{"enviar e-mail", ok, func() error { return nil }},
	}
	err := executarSaga(passos)
	fmt.Println("resultado:", err)
}

The output:

executando: reservar estoque
executando: cobrar cartão
executando: emitir nota fiscal
falhou em emitir nota fiscal: serviço fiscal fora do ar
compensando: cobrar cartão
  cobrança estornada
compensando: reservar estoque
  estoque liberado
resultado: serviço fiscal fora do ar

Two observations matter more than the code itself. The first is that the compensation can also fail, because it is also a call to a service. So the compensation needs retries and idempotency, like any other consumer. The second is that not everything can be undone. The email that was sent stays sent, and the customer who received the confirmation will see the cancellation notice and want to know why. In such cases, the compensation is a new business action, such as sending a cancellation notice. It records the correction in the history instead of erasing what happened. This difference between undoing and compensating is what confuses most people new to the pattern. The Saga Pattern note has the implementation view in more detail.

9. Event sourcing overview

Event sourcing reverses the traditional way of storing data. In a common system, the accounts table stores the current balance, and each movement overwrites the previous number. In event sourcing, the sequence of facts is stored (deposited 100, withdrew 30), and the balance is calculated by adding those facts. Martin Fowler described the pattern in an article dated December 12, 2005. In a free translation of the article’s opening sentence, the pattern captures all changes to an application’s state as a sequence of events. The ordered record of these events is called the event store.

This model has concrete gains. Auditing comes for free, because the history is the data itself. An old bug can be investigated by replaying the sequence up to the moment the problem appeared, and a new balance can be calculated in a different way without losing anything. The costs are also concrete. Queries that the fact model does not answer directly require projections, which are read models derived from the events and kept apart from writes, a separation that usually appears together with CQRS. Changing the format of an event requires care, because the old events remain in the history. And rebuilding the state from millions of events is slow, which leads to snapshots: a picture of the state at a certain point, so that the rebuild starts from there.

The example below implements the core of the model. The withdrawal is accepted only if the event version is the next expected one, which also settles the ordering question from section 6.

package main

import (
	"errors"
	"fmt"
)

type Evento struct {
	Versao int
	Tipo   string
	Valor  int
}

type Conta struct {
	Saldo  int
	Versao int
}

func (c *Conta) aplicar(e Evento) {
	switch e.Tipo {
	case "Depositado":
		c.Saldo += e.Valor
	case "Sacado":
		c.Saldo -= e.Valor
	}
	c.Versao = e.Versao
}

// Reconstrói o estado do zero, somando os fatos na ordem em que aconteceram.
func reconstruir(historico []Evento) Conta {
	var c Conta
	for _, e := range historico {
		c.aplicar(e)
	}
	return c
}

var ErrVersaoFora = errors.New("versão fora de ordem")

// Anexa só se a versão for a próxima esperada.
func anexar(historico []Evento, e Evento) ([]Evento, error) {
	if e.Versao != len(historico)+1 {
		return historico, fmt.Errorf("%w: esperava %d, veio %d", ErrVersaoFora, len(historico)+1, e.Versao)
	}
	return append(historico, e), nil
}

func main() {
	var historico []Evento
	var err error

	historico, err = anexar(historico, Evento{1, "Depositado", 100})
	fmt.Println("v1:", err)

	historico, err = anexar(historico, Evento{2, "Sacado", 30})
	fmt.Println("v2:", err)

	_, err = anexar(historico, Evento{2, "Sacado", 50})
	fmt.Println("v2 de novo:", err)

	historico, err = anexar(historico, Evento{3, "Depositado", 5})
	fmt.Println("v3:", err)

	c := reconstruir(historico)
	fmt.Printf("saldo reconstruído: %d (versão %d)\n", c.Saldo, c.Versao)
}

The output:

v1: <nil>
v2: <nil>
v2 de novo: versão fora de ordem: esperava 3, veio 2
v3: <nil>
saldo reconstruído: 75 (versão 3)

The append function refuses the second event of version 2, because the expected version was already 3. This is the same optimistic concurrency control that databases use for two concurrent writes to the same row. Figure 10 shows the path from a command to a projection and a snapshot.

In figure 10, the withdrawal command is validated and written to the event store as the Sacado v3 event. The balance projection is rebuilt by replay, reading the events from the store. The snapshot is a picture of the balance at version 2, and the projection can start from there instead of reading from the beginning. The dotted arrows show this shortcut, which is only an optimization: the source of truth remains the history.

Event sourcing combines well with event-driven architecture, because the events the system already publishes can be the history itself. But it is a choice that changes the mental model of the whole team. Someone who only wanted to store the balance will find that they need to rebuild it often, and the first time it happens it looks like just a sum. Understanding this early is usually cheaper than discovering it in the middle of an incident.

10. Concerns when scaling

Scaling an event-driven system means, in practice, dealing with the parts that do not show up in a two-line example. The concerns below are the ones that appear first in production.

Partitions and hot keys

The key that distributes messages across partitions can concentrate traffic. If a single customer generates most of the events, their partition becomes overloaded and the others sit idle. This is a hot key case. The solution depends on how much the order of that entity matters. Sometimes a composite key is enough (customer and day, for example). When order does not matter, a random key spreads the load, at the cost of losing the order guarantee for that entity.

Lagging consumers

Lag is the distance between the last published event and the last event processed by a consumer group. A growing lag means the consumer cannot keep up with the producer, and the bottleneck is usually in processing. It is the alert that matters most, because a growing queue is silent: the system keeps responding, and the email only arrives hours later. That is why observing the lag belongs in the plan from day one, not after the first incident.

Contract evolution

A published event becomes a contract with all consumers, including those that do not exist yet. Renaming a field or changing its type breaks whoever already reads the old format. The common practice is to version the contract, add new fields as optional and keep the old format published until all consumers have migrated. Replaying old events requires the same care, because the consumer has to understand an event written years ago.

Poison messages and DLQ

A poison message is one that always fails, because of invalid data or a bug that appears only with that content. Without handling, it is delivered again and again, blocking the consumer. The usual solution is to limit the attempts and, once they are exhausted, move the message to a dead letter queue (DLQ). Figure 11 shows this path.

In figure 11, the consumer tries to process the event. If the attempt succeeds, the message is acked and leaves the queue. If it fails, the diamond asks whether there are attempts left, and the dotted arrow returns to the consumer for a new attempt, with a growing wait between them. When the attempts run out, the message goes to the DLQ, which has an alert and a process to reprocess messages after the defect is fixed. Without this exit, the bad message blocks the whole queue, and nobody notices while the lag grows.

Observability

A flow that passes through five services and two brokers can only be investigated if each event carries a correlation identifier, copied from one service to the next. With it, it is possible to rebuild the path of an order from the logs of all the components. Distributed tracing, described in the Distributed tracing and OpenTelemetry note, automates this task. Without it, investigating a lost order becomes archaeology.

11. Common mistakes and good practices

The first mistake is treating a command as an event. The symptom appears when the event name says what someone else should do, such as EnviarEmail or AtualizarEstoque. The reason is that the producer keeps depending on whoever reacts, and the architecture loses the benefit that motivated the choice. The practice is to name the event after the fact, in the past tense, and leave the action to whoever consumes it.

The second mistake is assuming that delivery happens exactly once. Whoever writes the consumer with that assumption discovers the problem on a weekend, with a duplicated charge and a log that looks normal. The practice is to treat every delivery as possibly repeated, and to keep the table of processed events in the same transaction as the effect, as in section 5.

The third mistake is dual write without an outbox. It works in tests, because nobody kills the process between two commands, and it fails in production at the most inconvenient moment. The practice is to write the event in the same transaction as the data and let a relay publish it afterwards.

The fourth mistake is compensation without retry. A saga that compensates once and gives up leaves the system half undone, with stock reserved for a cancelled order. The practice is to treat the compensation as a step that must be idempotent and that has its own failure handling.

The fifth mistake is assuming global order. A local test with a single consumer passes, and in production two consumers compete for the same entity. The practice is to find the smallest order the domain really requires, choose the partition key based on it, and make the handlers tolerant of the rest.

The sixth mistake is publishing events without a version. The first year looks fine, and in the second year a new field breaks three consumers that nobody remembered having. The practice is to put the contract version in the event itself from the start, even when it seems excessive.

The seventh mistake is the system where everything is an event. Each service reacts to the previous one, and no diagram shows the complete flow. When an order gets stuck, the team spends a week finding out at which of fifteen consumers it stopped. The practice is to use orchestration when the business flow must be visible, and choreography when the reactions are truly independent.

12. When not to use it

Not every problem calls for event-driven architecture. When the user needs an immediate answer that depends on the result, such as validating a card before confirming the purchase or showing a balance on the screen, the flow is naturally synchronous, and forcing it to be asynchronous creates a waiting screen that nobody asked for. When the volume is low and a local transaction in the same database solves the problem, the broker is one more piece to maintain, monitor and update, with no proportional benefit. When the team is small and has no experience with the problems in sections 5 to 8, the learning cost may outweigh the gain for quite a while.

It is also worth considering that debugging an asynchronous flow is harder. The error happens at another moment and in another process, and the call stack a developer uses to understand the code stops existing. This difficulty is paid for with observability and documentation, and whoever did not plan for it feels the difference at the first incident.

Exercises

1. Take the example from section 1 and rewrite the order creation with PedidoCriado in place of the email call. List the information the event must carry so that the email consumer does not need to call the order service back.
2. Change the consumer from section 5 to record the event ID in a map separate from the balance, without the same transaction. Find a scenario, described in text, in which the balance ends up wrong even with the record. Then explain why the same transaction solves that scenario.
3. In the outbox example, change the relay to publish in batches, marking the rows only after all of them are published. Think about what happens if the crash occurs in the middle of the batch and how many events may be repeated.
4. Add a fifth step to the saga in section 8, such as recording the sale in an accounting system, and a compensation for it. Then make the failure occur at step 5 and check the order of the compensations in the output.
5. Choose a feature of a system you know and decide whether it needs a queue, a log, a saga or none of these. Write the justification in one paragraph, citing what would happen if the choice were wrong.

Glossary

Coupling: the degree to which a component depends on another component working.

Ack: confirmation sent by the consumer to the broker, saying that the message was processed.

At-least-once: delivery guarantee in which the message is never lost, but may arrive more than once.

At-most-once: delivery guarantee in which the message is delivered at most once, without repetition, but may be lost.

Broker: intermediary that receives, stores and delivers messages between producers and consumers.

Compensation: action that undoes, or neutralizes in the business sense, a completed step of a saga.

DLQ: dead letter queue, where the messages that failed after the configured attempts go.

Event: immutable record of a fact that has already happened.

Event store: storage that keeps events in order, used as the source of truth in event sourcing.

Idempotency: property of an operation whose effect does not change when it is repeated.

Lag: distance between the last published event and the last event processed by a consumer.

Log: sequence of messages, retained for a period, in which each reader keeps its own position.

Offset: position of a message within a log partition.

Outbox: table in which the event is written in the same transaction as the data, to be published later by a relay.

Partition: subdivision of a topic in a log, in which order is guaranteed.

Projection: state derived from a sequence of events, computed to make queries easier.

Relay: process that reads the outbox and publishes the pending events to the broker.

Saga: sequence of local transactions with compensations, used to keep consistency between services without a global transaction.

Snapshot: picture of a state at a point in the history, used to speed up the rebuild in event sourcing.

Sources

Accessed on November 2, 2026:

• Apache Kafka, producer configuration (enable.idempotence defaults to true, and acks defaults to all): https://kafka.apache.org/41/configuration/producer-configs/
• Apache Kafka, version 3.0 upgrade notes (producer idempotence enabled by default, and the bug that prevented the default in 3.0.0 and 3.1.0): https://kafka.apache.org/30/getting-started/upgrade/
• Apache Kafka, downloads page (list of 4.x versions and release dates): https://kafka.apache.org/community/downloads/
• AWS, Exactly-once processing in Amazon SQS (5-minute deduplication window in FIFO queues): https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html
• Martin Fowler, Event Sourcing (published on December 12, 2005; the definition cited is in the opening of the article): https://martinfowler.com/eaaDev/EventSourcing.html
• Chris Richardson, Pattern: Transactional outbox (dual write problem, two forms of relay and at-least-once guarantee): https://microservices.io/patterns/data/transactional-outbox.html
• Hector Garcia-Molina and Kenneth Salem, “Sagas”, SIGMOD ‘87, pages 249 to 259, DOI 10.1145/38713.38742. Checked through the bibliographic record in search results, without reading the full text.
• Go, toolchain go1.27.2 (darwin/arm64), used to run and check the five examples of this lesson.

Not verified in this review: Kafka’s default retention, the details of Kafka transactions for exactly-once across topics, and the internal implementation of CDC with Debezium and the other tools mentioned in passing. These points are basic knowledge of the field and were not checked against an official page in this review. Designing Data-Intensive Applications, by Martin Kleppmann, is the recommended reading to go deeper into sections 5 and 10, and it was not consulted in this review.

Related

• Idempotency
• Saga Pattern
• Publish/subscribe
• Kafka
• What Kafka is and how it works
• Fundamentals of messaging and distributed systems
• Practical messaging with RabbitMQ and Kafka
• NATS
• Transaction
• Resilience with circuit breaker and retry
• Distributed tracing and OpenTelemetry
• Hexagonal architecture