Provenance before analysis: the register and what it excludes

The third principle of the Ethical Technology Charter is provenance of data, and a principle of that kind is either an entry in a register or an aspiration. The division keeps a provenance register with one entry per dataset. Each entry records where the data came from, the chain of licence from the original collector, the lawful basis for holding it, the purposes for which it may be used, the deletion date, and the named person accountable for it. The register held 22 datasets in April, against 14 a year ago, and it is the gate through which every input to every model passes.

The register exists because the alternative fails quietly. An analyst who obtains a useful file from a broker, a conference or a former colleague can build something valuable with it in a week. The problem surfaces two years later, when a client asks what the analysis rests on, or a counterparty asserts rights over the underlying records, and nobody can answer. The cost of answering late is not a fine. It is the withdrawal of a product that clients have come to rely on, and the loss of the work built on top of it.

Nine datasets offered to the division in the past twelve months were declined. Three could not demonstrate a chain of licence back to the original collector: the seller had rights to distribute, but could not show that the party it bought from had rights to sell. Two contained personal data about farm operators and clinicians that no mandate of ours requires, which the fourth principle of the Charter puts beyond use. Four carried terms permitting analysis but prohibiting use as an input to a model, a restriction that appears in more commercial licences each year and is frequently missed by the people who sign them.

Declining a dataset has a price and it should be stated. One of the nine was a price series for animal-health products across four countries that would have improved one of our client analytics materially. The division rebuilt an equivalent series from its own origination records, from distributor invoices held under facilities the Commercial Finance division writes and from published tender results. That work took five months and cost more than the licence would have. It produced a series the firm owns outright, can document, and can license onward without seeking permission from anyone.

Provenance also governs what a dataset may be joined to. Two datasets that are individually unobjectionable can, when combined, identify an individual holding or a named operator. The register therefore records permitted joins as well as permitted purposes, and any new join is reviewed before it runs rather than after the output is circulated. Four proposed joins have been refused since the register opened. In three of them the fix was aggregation to a coarser geography, which cost that analytic very little and removed the question entirely.

The rule the division works to is short. No dataset enters a model without a register entry, no register entry is made without a documented origin and lawful basis, and no dataset is used for a purpose the entry does not name. This is slower than the practice of most data businesses, and it is the reason the firm sells analytics to institutions that ask hard questions in their own diligence. Provenance is not a compliance overhead in this division. It is the product specification.

Published 2023-04-18 by the Information & Data division. Research is prepared for eligible counterparties and does not constitute advice.