Data Change Management
Every organization keeps records of things: customers, suppliers, products, contracts, locations. I call them information objects. The records go out of date sooner than most teams expect, because the things they describe keep changing:
- A company changes its name, merges, splits or goes bankrupt.
- A product gets a new version, a new price or a new owner.
- A site is opened, converted to another use or closed.
- A contract changes hands, and a contact person moves to another employer.
How do you manage these changes? What about the changes nobody notices, or notices a year later? How many active suppliers does the company have today, how many did it have five years ago, and how many will it have next year? What if two sources say different things about the same object? And who decides when a lead counts as a customer?
What stale data costs
Change never stops. Sometimes it happens abruptly, mostly it happens slowly, and it follows similar patterns in every field I have worked in. Most of the tools that IT uses for it were built for data that is entered once and then read. A company that does not automate the handling of change works with incomplete, outdated or wrong information. Providers that sell such information as market data frustrate the customers who pay for it: the quality is poor, and much of the data is missing.
What databases leave out
Data change management is my name for treating change as part of the data. A change is as important as the value it replaces. Databases are mostly used to hold the best available state as of today, for the party that manages the data. They are rarely used to
- change a schema without programming,
- version every change to every record,
- let the people who know better update, rate and correct the data,
- show the data as it stood at another time than today.
These four points are the core of data change management. A change is recorded and shown as it was best known at a given time and from a given point of view.
Identity without a static key
One result of this work concerns the exchange of data. Static primary keys are the common way to exchange records between systems. They do not meet the requirements of change, because nearly every piece of information about an object can change at any time. Reliable exchange needs a system that can let go of static IDs and offers other ways to identify and match records.
The concept behind it
I developed the concept from long observation of how change happens and from ideas of evolutionary research. The store replaces the usual set of database tables with small units modeled on genetics. A nucleotide holds one pair of data about an object: a value with its type, its source and the commit that wrote it. A strand in the manner of RNA, the simple carrier, stores the data. A strand in the manner of DNA, the complex one, holds the processing: triggers, processes and logic.
A radically simple storage layer allows any complexity in the objects that evolve on it. Storage was the easy part of the project. The first set of functions was hard, because some components are needed to make the system work, and they depend on other components of the same system. Organisms have the same problem: there has to be an origin.
Results
The aim of the research project was reliable data that organizes itself. The project is complete. Its findings run in production on an information platform, and I develop them further there. Every change to a record is a version. Every fact carries its source, the time it was observed and the time it was true, so the platform answers what was known at any date, and from whom. The patterns of change are similar across industries, so the model fits insurance, finance and every other field with long-lived objects. The Research pages describe the current state part by part.