CDC – change data capture – aims to identify changed data to enable actions on those [1].
The (transactional) outbox pattern is about service commands that update a database and send messages/events [2].
That is so similar or even the same, what are the differences? Let’s look into what the CDC and the outbox pattern is about in more detail.
CDC – Change Data Capture
CDC is a collection of patterns. Changed database data are identified in multiple ways including
- timestamps
- versions
- status indicators
In case data in a database change, the changed data are identified e.g. in a database (materialized) view. A message relay forwards these as change events to a message broker. With that setup consumers can consume events from the message broker.
Alternatively it externalizes the transaction log of a database as a stream of events to interested consumers.
Outbox Pattern
The outbox table is a key concept of the outbox pattern. Changes in the service’s database are stored in an outbox table. The transactional outbox pattern specialization ensures the transaction is only complete with adding an item about the change to the outbox table.
The transaction log of a database can be used to add items to the outbox table. This way the database transaction does not need to include the outbox table addings.
Changes in the outbox table are read from a message relay that produces change events to a message broker.
The outbox table can be ephemeral – messages read by the message relay cause the read item removal in the outbox table.
The outbox table can be append only as well. This way it represents a source of truth for all database change events.
Conclusion
CDC and Outbox Pattern serve often the same purpose. Database changes are read by a message relay and sent to a message broker.
CDC and Outbox Pattern are different in how the message relay reads change data.
The Outbox Pattern implements an additional outbox table that contains the changed events
CDC uses database (materialized) view(s) from which changes are read.
While the additional outbox table adds additional complexity to manage change events, it’s also an additional database table to store change events of a database.
Links
[1] https://en.wikipedia.org/wiki/Change_data_capture
[2] https://microservices.io/patterns/data/transactional-outbox.html
CDC may look leaner at first sight, but it is actually more complex and brittle. It usually relies on internal Wal log which may grow so big it renders the whole database unusable due to CDC reader/ subscriber failing to read from it on time. Outbox pattern is more verbose but thanks to its targetedness it is easier to keep under control and helps avoid blast radius spill over your whole DB instance.