What counts as a model: building the inventory

Model risk work usually begins with validation. It should begin with counting. The policy that took effect on 1 February defines a model as any method applying statistical, mathematical or algorithmic technique to input data to produce an estimate or a ranked output on which the firm may act. That definition is deliberately wide. Under it, the first sweep across the group found 38 models, of which 11 were spreadsheets maintained by an individual and 6 were embedded in services bought from vendors. Only 21 were the kind of thing a quantitative team would have listed unprompted.

The spreadsheets were the finding that mattered. A workbook that prices an unlisted position for the valuation committee performs exactly the function of a valuation model, and it carries risks the coded models do not: no version control, no independent review, no record of the assumption that was changed at the request of a desk. Four of the 11 have been rewritten into the reviewed code base, five have been brought under version control with a documented owner and a change log, and two were retired because the position they priced had been sold.

Tiering decides how much work each model receives, and the policy tiers by the cost of being wrong rather than by technical complexity. Tier 1 output directly determines an order, a reported valuation, a capital or liquidity decision, or a screening outcome for a client. Tier 2 informs a human decision without determining it. Tier 3 produces internal management information with no external effect. Seven models are Tier 1 today. A simple interest calculation embedded in a treasury sheet is Tier 1. A large machine-learning system that ranks research topics is Tier 2. Complexity and consequence are different axes, and the policy follows consequence.

Validation for Tier 1 is independent of the developer, is completed before first use and is repeated every year. It has three parts: a review of the conceptual approach and its documented limitations, an outcome analysis against realised results, and a challenger built by the validator on a different method. The challenger is the part teams resist and the part that finds errors. In the first cycle, a challenger built for a valuation model on an unlisted credit position produced values 6 to 9 per cent lower in illiquid months, and the difference traced to a liquidity adjustment applied at the wrong point in the calculation.

Monitoring is what keeps the inventory alive after the validation is filed. Every Tier 1 model is monitored against performance thresholds set at validation, and a breach requires the owner to act within a defined period. Every Tier 1 model can be stopped inside fifteen minutes by its named owner, by the Chief Risk Officer or by the Head of Technology and Ethics. That control has been exercised once, on a signal whose input feed changed format without notice. The stop worked. The notification that should have preceded it did not, and the vendor contract has since been amended.

The inventory will hold about 44 models by the end of the year, nine of them Tier 1, and it will keep growing as the Information & Data division builds out its analytics. Growth is not the measure of success here. The measure is that each entry has a person against it, a tier, a validation date and a link to the provenance records of its inputs. Human accountability, the first principle of the Charter, is not a statement of intent in this division. It is a column in a register, and it is populated.

Published 2023-06-20 by the Quantitative Strategies division. Research is prepared for eligible counterparties and does not constitute advice.