In the decade since US banking regulators published their seminal guidance on model-risk management (OCC Bulletin 2011-12 and SR Letter 11-7), the development, monitoring, testing, and validation of models have evolved considerably. While earlier rules focused narrowly on validation, SR 11-7 introduced a more comprehensive approach. In particular, the guidance addressed the potential for erroneous model outputs, taking into account errors originating between design and implementation. It also created an expectation that decision makers would understand model limitations and avoid using models at odds with their original purposes.
Against this background, many financial institutions have developed highly structured model-risk-management frameworks, enabling them to manage large numbers of models and multiple risks. Still, the events of the past year have shown that risk management is not the same as risk elimination. The COVID-19 pandemic and its economic impacts have created uncertainty around model performance, particularly in credit and stress testing. In many cases, this has led to the need for ad hoc managerial overlays. It has also highlighted the imperative that model-risk management enables a rapid response to an evolving external environment.
As a result of the COVID-19 crisis, financial institutions find themselves with an array of models that need to be recalibrated or rebuilt and, subsequently, validated, monitored, and managed. The three lines of defense—model-development ownership or sponsorship, risk validation and compliance, and auditing—are all facing substantial workloads. At the same time, financial institutions must decide whether to include data from the COVID-19 period and how to account for the impact of government interventions. Furthermore, if they proceed via existing approaches, they are likely to see extra costs and delays in bringing models to market. Further compounding these challenges, developers and validators are already burdened by expectations for risk management of new AI- and machine-learning-based models.
While the challenges associated with these dynamics are significant, they can be mitigated if banks apply the model-risk-management lessons that have been learned over the past ten years. In our view, that means targeting the best practices in six key areas: strategy, process enhancement, agile modeling, talent and culture, data architecture and infrastructure, and technology. If implemented to their potential, these six pillars may allow banks to rebuild better after the crisis and avoid undesirable trade-offs between cost, timelines, and quality.
1. Optimize model strategy
In its most basic form, model strategy can be optimized via two tools: inventory rationalization and risk-based prioritization. Rationalization involves the identification of model synergies across the portfolio and smart allocation of resources to reflect model needs. Banks that have not recently rationalized their inventories may discover that similar models have been developed at different times in various businesses. One large financial institution found that numerous individual analysts had developed cash-flow models, which generated hundreds of models whose differences were immaterial to business outcomes. An initiative to consolidate models and guide future development resulted in significant savings and better management of model risk.
Rationalization is not just a case of decommissioning duplicates. For instance, many banks are converging on a single set of models for comprehensive capital analysis and review (CCAR) and financial planning, and some modeling teams will need to start from scratch. However, while there may be costs involved, they will be outweighed by the benefits in terms of coherence, decision making, and reduced model-management liabilities over the longer term.
Risk-based prioritization implies the allocation of more resources to riskier activities. Under current practice, most banks assess risk through a tiering process, which generally reflects a combination of model complexity and materiality. The exercise helps banks focus efforts on the most critical risks, as well as the breadth, depth, priority, and frequency of validation activities.
2. Enhance processes
The majority of models are built with a standardized development and validation process. Still, many banks could go further, making model-development and model-validation guidelines more responsive to use cases (business types) and more focused on conceptual soundness while avoiding a laundry-list approach to testing. Best practice is to align templates to these standards, which can be customized by use-case and modeling technique. The right standards and templates will enable automation of tasks in development, documentation, and validation and will also facilitate audits.
Some financial institutions have formalized interactions between the first and second lines of defense for model development and validation, creating requirements for submission, as well as timelines for validation. Occasionally, lines of defense have entered into service-level agreements with each other.
3. Adopt agile modeling
Agile modeling accelerates development and validation where there are significant uncertainties (for example, where use cases have not been modeled previously) that could slow the transition from ideation to production and where the use case has high materiality. More generally, the use of agile modeling is guided by business needs.
Agile approaches are characterized by the close involvement of key stakeholders, such as subject-matter experts from the business or risk functions, as the model is being developed. The validation team, meanwhile, can engage with the modeling team throughout the development process, leveraging tollgates to carry out regular reviews.
In the archetypal incarnation of agile, a minimal viable model is speedily used for A/B testing or put into production for business purposes. A simplified validation process could impose use restrictions for market testing, after which a more refined version of the model may follow. Of course, in practice, a mix of standard and agile approaches can be deployed, with features of either applied as necessary.
The implementation team is also more frequently and deeply involved in agile modeling. The team starts building data pipelines as soon as the development team identifies data elements or when the elements are submitted to validation for tollgate review. Similarly, when agile modeling is adopted, the implementation team starts coding the new model in the decision engine as it is submitted to validation. While the involvement of the implementation team in the agile approach can result in additional costs (any change made by model developers after validation findings may result in rework for the implementation team), its early involvement enables a much shorter time to market.
4. Align talent and culture
For optimal functioning of model-risk management across the three lines of defense, each line must have sufficient capacity and the right mix of capabilities and seniorities. Many financial institutions are now recruiting talent across the lines of defense, with profiles previously seen in one line now appearing elsewhere. The first line is continuing to hire data scientists and data engineers, but the second line is taking on former first-line bankers to validate qualitative approaches and expert judgements, and the third line is recruiting model developers to build quantitative tools for audits.
To facilitate model development, some institutions have built centers of excellence, while others have aligned modeling teams with business units. In the latter case, an “excellence review and challenge” unit can ensure sound modeling and consistency across teams. For validation, banks have generally been more consistent in their approaches, with centralized approaches prevailing. Newer ideas include splitting phases in model- development or model-validation processes (for example, ideation, data analysis, model testing, documentation) between offshore, nearshore, and onshore locations—aiming to align them with specific talent capabilities. In practice, this could mean that a bank would offshore data cleaning and model testing to leverage quant capabilities in a certain region while retaining model ideation and development onshore. In all cases, these strategies should be accompanied by a strong culture of model-risk management across the lines of defense.
5. Upgrade data architecture and infrastructure
Standardization, agile processes, and automation are contingent on data that are fit for purpose. Moreover, to fully realize cost and time-to-market benefits, there is a need for appropriate data governance, quality testing, and architecture, as well as infrastructure. Some leading banks have deployed a common data source as a platform for developers and validators so that data lineage is guaranteed. These platforms are now often associated with distributed-computing architectures and, in some case, reside in the cloud.
6. Embrace automation
Automation delivers its benefits synergistically with process standardization and maturity of data architecture and infrastructure. It can be applied to all stages of model management—including development, testing, validation, implementation, use testing, monitoring, and maintenance. However, in spite of sometimes significant investment, many financial institutions fail to unlock the potential efficiency gains. A more integrated, strategic approach to managing the model life cycle would involve end-to-end automation across development, validation, and monitoring. Introducing a more robust, easy-to-use, and comprehensive infrastructure will avoid the bottlenecks that slow communication between model development and validation. Financial institutions at the forefront of these efforts often take three distinct steps: define the target state, identify current pain points, and outline design principles to implement a model-excellence workspace.
The COVID-19 pandemic has created significant challenges for financial institutions in both modeling and model-risk management. While the principles of the ten-year-old SR 11-7 continue to apply, the time is ripe for new approaches that reflect and extend its ideas. A model-risk-management program in which the three lines of defense embrace the six levers discussed here will lead to significant gains in terms of cost, timelines, and model quality. These, in turn, will catalyze superior model-risk management, increased institutional resilience, and closer alignment with the regulatory agenda.
