Evolving data

Data is usually the hardest part of a system to change, because it outlives code and is often shared. Pramod Sadalage, who joined for the 2nd edition, co-wrote Refactoring Databases with Scott Ambler, and the book carries those ideas over.

Practices

• Migrations as code. Schema changes are versioned scripts in source control and run by the pipeline (Flyway, Liquibase, Alembic, and so on). Nobody edits a production schema by hand.
• Expand and contract. For a breaking change (rename a column, split a table):
  1. Expand: add the new structure alongside the old.
  2. Migrate: write to both, backfill, move readers to the new form.
  3. Contract: remove the old structure once nothing uses it.
• Each quantum owns its data. Shared databases couple services at the strongest level. Integrate through APIs or events instead.
• Avoid integration databases, where several applications read and write the same tables.
• Age and inappropriate data entanglement: watch for data being used for things it wasn’t designed for, such as reporting queries hitting transactional tables.

Cloud angle

• Managed databases make provisioning cheap, which makes “one data store per quantum” more practical.
• Event streams (Kafka, Kinesis, Event Hubs, Pub/Sub) let you publish data for others without sharing the store.
• Fitness function ideas: “no service connects to another service’s database” (check network rules or IAM), “all schema changes go through the migration tool” (check for drift).