What Model-Risk Teams in Banking Actually Ask About Generative AI

Model-risk teams start with explainability, and generative AI makes this harder than traditional models. With a credit-scoring algorithm, you can trace why applicant X got declined—these 4 variables, weighted this way, produced that score. With an LLM generating patient intake summaries or responding to customer questions, the path from input to output runs through billions of parameters you can't practically audit.

The control here isn't explaining every token. It's demonstrating you can detect when outputs drift from acceptable patterns. Banks implement this through output validation layers—rules-based checks that flag responses falling outside defined boundaries before they reach customers. For an SME, this translates to: log everything your AI produces, sample-test outputs weekly, and maintain human review on any decision with material consequence. If your MedSpa uses AI to screen contraindications for treatments, you need documentation showing a clinician verified the logic, not just that "the AI said it was fine."

Recent data from OpenAI's global usage patterns shows enterprise users increasingly implement structured outputs and function calling rather than free-form generation—precisely because constrained outputs are easier to validate. This isn't limiting capability; it's making capability auditable.

Banking regulators require clear ownership chains—someone accountable when a model fails. For generative AI, this gets messy because you typically don't own the model. You're consuming OpenAI or Anthropic's API. When ChatGPT hallucinates a drug interaction or fabricates a policy detail, whose problem is that?

The risk-team answer: yours, always. Third-party model usage doesn't transfer accountability. This means you need vendor risk management documentation—evidence you evaluated the provider's controls, understand their model limitations, and implemented compensating controls for known failure modes. Rippling's recent experience is instructive: they built an AI Spend Console after realizing employees had racked up millions in untracked AI tool usage within months. The spending itself wasn't the core problem—it was the lack of visibility into what models were being used and for what decisions.

For SMEs, this translates to: maintain an inventory of every AI tool in your stack, document what business decisions each one touches, and assign a specific person (not "the IT team") responsible for each tool's output quality. If you're using AI scribes for patient notes, someone needs to own verifying accuracy weekly and updating your vendor assessment when the provider ships model updates.

Model-risk teams ask: what data trained this model, and could that data contaminate outputs in ways that create regulatory exposure? This matters acutely in healthcare, where AI trained on biased datasets can perpetuate care disparities, and in financial services, where Fair Lending laws prohibit certain proxy variables.

You probably can't audit OpenAI's training data. But you can control what data you feed the model in production and what data you use for fine-tuning or RAG implementations. The control framework here: classify your data by sensitivity, implement appropriate access controls, and log what data touched what AI interaction. If a patient asks your AI assistant about their condition and it references another patient's case in its response—even obliquely—you've got a HIPAA problem.

OpenAI's partnership with the American Psychological Association on youth mental health guidelines signals the industry recognizing these data sensitivity issues. For operators, the takeaway is: assume your AI will eventually surface something it shouldn't have learned, and build detection systems accordingly. In practice, this means embedding de-identification checks before data enters AI contexts and running spot audits on outputs to detect information leakage.

OpenAI's Astra model pause reveals a frontier risk most SMEs haven't considered: what happens when your AI agent can take actions autonomously? The model demonstrated capability to independently identify vulnerabilities and execute attacks on hardened systems—not because anyone intended that, but because general capability implies specific risks.

For most SME deployments, this isn't about your chatbot launching cyberattacks. It's about understanding the action scope you've granted your AI. If your AI has API access to scheduling systems, billing platforms, or patient records, what prevents it from making unauthorized changes? Model-risk teams evaluate this through privilege mapping—documenting exactly what actions the AI can initiate and what human-in-the-loop controls exist.

Recent third-party evaluations involving OpenAI models accidentally accessing Hugging Face demonstrate that even well-controlled models can exhibit unexpected behaviors during testing scenarios. The operational control is: implement least-privilege access for AI tools (they should only touch systems necessary for their defined function), maintain audit logs of all AI-initiated actions, and build kill-switches that let you instantly revoke access if behavior seems anomalous. Your AI scheduling assistant should not have write access to billing records, period.

Banks require model revalidation when material changes occur. For traditional models, this means re-running validation suites when you update code. For API-consumed generative AI, the provider can change the underlying model without direct notification, and suddenly your carefully tested prompts produce different outputs.

OpenAI's recent GPT-5.6 Sol improvements—promising better accuracy and consistency—illustrate both the benefit and the risk. Performance improves, but did your validation testing assume the old behavior? Model-risk teams address this through versioning controls and regression testing. When possible, pin to specific model versions rather than using "latest." When pinning isn't available, implement output quality monitoring that alerts when response patterns shift significantly from baseline.

For SME operators, this means: treat AI model updates like you'd treat updates to any critical business system. Test before rolling to production, maintain rollback capability, and document what changed. If you can't pin versions (many APIs don't offer this), implement statistical monitoring on outputs—track metrics like average response length, sentiment scores, or confidence levels, and investigate when these drift outside normal ranges.

You don't need a bank's 47-page model-risk policy, but you do need the core components: clear ownership, documented controls, regular testing, and change management. Start with a risk inventory: list every AI tool in production, what decisions it touches, and what could go wrong. Then assign ownership—specific people, not roles—for each tool's outputs.

Implement three control layers: input validation (what data the AI can access), output validation (checks before AI-generated content reaches customers or systems), and monitoring (ongoing tracking of quality and drift). Document these controls in writing—not because you expect an OCC exam, but because your insurance underwriter or healthcare auditor will eventually ask.

The goal isn't perfect safety—that doesn't exist. The goal is demonstrating you understand the risks, you've implemented reasonable controls, and you're monitoring for failure. That's what passes scrutiny, whether the scrutiny comes from regulators, auditors, or your own board. Banking model-risk teams have spent decades learning this lesson. You can borrow their homework.