Change management, de-duplication and workflows

The other research pages describe a store at rest. In daily work the store changes: imports put new records beside existing ones, editors correct facts, and the schema gains or loses a field. These are the rules I put around those changes on an information platform.

Same address, different company

Two companies can be registered at one address, and two products from different makers can carry the same name. A matcher scores both pairs high, and a merge that trusts the score destroys one record.

My rule is a set of identity vetoes that every scanner and merge routine checks before it writes. A veto is a hard contradiction between two records. Lifecycle status is one: a dissolved company and a company founded later at the same address are two entities. A designator is another: “Plant North” and “Plant South” are siblings. So is a measured attribute outside a fixed tolerance.

A high match score never overrides a veto. A shared name, address or external identifier generates a candidate and decides nothing. One identity for an object across organizations describes the identity model behind this rule.

A merge is a soft delete. The keeper inherits the identifiers of the retired record, and the retired name becomes an alias, so a wrong merge can be taken back.

What a duplicate scan has to measure

A scan reports the pairs it declined as well as the merges it made: how many candidates each signal generated, how many fell to a veto, and which pairs stayed below the automatic threshold.

Candidates cannot come from names alone. The English and the French name of one company share no token, so a scan also takes candidates by location and by agreement of measured attributes.

A checked sample belongs in every report. When a manual check finds distinct entities among pairs that passed a strict filter, the signal is too weak for automatic merges. Pairs the vetoes cannot settle go to AI agents under the same rules as humans, and “unsure” leaves both records in place.

Changes to facts

Every fact is a versioned row, as Knowledge kept with its time and its source describes. The row is the truth, and a flat column beside it is a projection. A failed row write is a failed save. Where rows and columns disagree in old data, I decide the correct side per field, because a blanket rule writes wrong values in bulk.

A rollback retires the rows that an import wrote, and the earlier rows show again. It deletes nothing. Saving the old value a second time would be a new edit and would leave the bad rows in the history.

Changes to the schema

A schema change starts with a count. Before a flag leaves the code, I count the rows that this flag alone hides, because they become visible when it is gone. Before a table goes out of use, a fixed sequence of checks proves that nothing reads or writes it. The table then gets a marker prefix and stays in PostgreSQL. The same counts run after the change and have to match the prediction.

Scripted workflows and writer caps

Scripts write through an SDK on top of the API, the path of an editor’s save. The SDK journals every call without the access token. Each environment has a cap on concurrent PostgreSQL writers. When a job changes its writer count, all writers stop first, and the remaining rows are split into one shard per new writer.

Open questions

The vetoes, the merge without deletion, rollbacks that retire rows, counted schema changes and the writer cap are in production use on a client’s information platform. Two questions are open.

The first is where automatic merging should stop. Below a threshold the matcher cannot tell a true duplicate from a close neighbor, and those candidates need a person or better evidence.

The second is how to check shared identifiers before a record is created, so that a duplicate is refused at the door and never has to be merged later.