Data Foundation

The data work that has to happen before the automation

Three systems hold the same field under three names, none of them agrees, and every automation built on top of them inherits the disagreement and turns it into a decision that nobody in the room can defend.

No model repairs that. A model only moves the disagreement along faster.

The work starts with a map: every field the business argues about, where each copy lives, and which system is allowed to be right. Definitions are written in plain language, an owner is named against each one, and the pipelines that move it are rebuilt in Airflow so the schedule is explicit.

None of that survives a quarter unless the plumbing says when it has broken:

• one definition per field, written in plain words

• a named owner for every source of record

• tests that run on the data, not just the code

• pipelines that fail loudly and stop early

Storage is the easy decision. Postgres for the modelled tables, object storage for whatever arrives raw, and a clear line between the two so that nothing is ever transformed in a place where the original can no longer be recovered.

The harder decision is how much history to keep and how honest to be about it. Records change meaning over time, and a table that overwrites yesterday’s value will answer a question about last quarter with today’s answer and give no sign that it did.

Only then is automation worth building. A model reading a layer with owners, definitions and tests behind it makes a decision that can be traced back to a row, which is the difference between a result and a guess.

Nobody enjoys paying for this stage. It is the stage that decides whether anything after it can be measured, and it is a great deal cheaper here than after launch.