Making Sure Your Models Work

2-6-15-Crowe.pngAs the financial institutions industry evolves, banks increasingly rely on complex models to support their economic, financial and compliance decision-making processes. These models are developed to represent real-world relationships. Because the models are simplified representations of reality, however, they present inherent risk to an institution if they leave out vital information and, as such, should be validated to make sure they meet the financial institution’s objectives. The complexity of a financial institution’s structure—which could include multiple legal entities and business units and a growing breadth of products and services—creates additional risk and increases the need to manage risk at an enterprise-wide level. By implementing a model risk management framework that includes model validation, financial institutions can mitigate the risks they face.

Model Risk Management Framework
Federal banking regulators have issued guidance called the “Supervisory Guidance on Model Risk Management (OCC 2011-12 and SR 11-7) dated April 4, 2011. How does an institution begin to develop an effective model risk management program? 

To address this question, financial institutions should oversee a comprehensive model risk management framework that governs the development, implementation, and ongoing use of bank models. A prescriptive framework better allows banks to predict and identify risk more accurately and, therefore, make better top-level and line-of-business decisions based on model results.

The regulatory guidance provides banks with a basic framework for deploying an enterprise-wide model risk management program, and examiners now expect banks to use a framework when designing, implementing, and improving all models. Examiners are looking for a formalized, prescriptive methodology dictating the three fundamental components of the model risk management framework that the guidance identifies and requires: 1) model development, implementation, and use; 2) model validation; and 3) model governance. This article focuses on the model validation component.

Model Validation
Once the organization has inventoried and assessed the risk of the various models, ongoing evaluation is required to confirm that results are accurate and controls are adequate. The guidance defines model validation as “the set of processes and activities intended to verify that models are performing as expected, in line with their design objectives and business uses. It also identifies potential limitations and assumptions, and assesses their possible impact.”

Model validation confirms that an institution’s model is aligned with business and regulatory expectations and is properly executing the tasks that take into account the underlying risks and that lead to the bank accomplishing its business objectives.

The approach to validation focuses on the following four components related to the conceptual design, system, data and process for models:

  1. Conceptual design validation
    Do the conceptual design and capabilities of the model meet the identified business and regulatory needs? When assessing the conceptual soundness of a model, executives should consider the metrics and evidence supporting the ability of the model to predict risks accurately and execute the business objectives of the model. This process might include performing back-testing or outcome analytics to compare model outputs to corresponding actual outcomes or results.
  2. System validation
    All technology and automated systems implemented to support a model have inherent limitations. System validation independently confirms that the development, implementation and ongoing use of technology are properly designed and integrated enterprise-wide to support the model.
  3. Data validation
    Is the model capturing accurate and complete information? Data errors or irregularities likely impair model results and might lead to the organization failing to identify and respond to elevated risks or making poor decisions based on inaccurate model output.
  4. Process validation
    Model risks can occur even within well-developed models if poor decisions are made on model output or the output is poorly managed. It’s important to verify that the design and ongoing sustainability of the model’s processes are resulting in output that is accurate, managed effectively, and subject to appropriate controls.

As financial institutions work to adapt to increasingly stringent regulatory guidelines, the industry has adopted increasingly high standards for managing the risks inherent in models.  In order to meet the heightened expectations set by regulators, financial institutions must design, implement, test, and improve their models on an ongoing basis and within a sound model risk management framework. Institutions that do not adopt a documented, consistent model risk management framework risk increased regulatory scrutiny.

Chad Kellar


John Epperson