What Model-Risk Teams in Banking Actually Ask About Generative AI

The first question model-risk teams ask is deceptively simple: can you explain how the model produced a specific output? For traditional credit models or fraud detection systems, this means showing feature weights, decision trees, or logistic regression coefficients. For generative AI, the answer is significantly harder.

Large language models operate through billions of parameters and attention mechanisms that create emergent behavior. You can't point to a single feature and say 'this is why the model recommended this investment strategy.' OpenAI's recent work on coding evaluations highlights this challenge—even on structured tasks like software engineering benchmarks, determining why a model succeeded or failed requires careful analysis of confounding factors and benchmark quality itself.

Model-risk officers know that regulators like the OCC and Federal Reserve have published guidance requiring explainability for models used in credit decisions, risk management, and consumer-facing applications. If your generative AI system denies a loan application or recommends portfolio adjustments, 'the model said so' isn't sufficient documentation. Operators should prepare to show: prompt engineering specifics, retrieval-augmented generation workflows that tie outputs to source documents, and deterministic business-rule layers that constrain model outputs to acceptable ranges.

The second interrogation point is data lineage. Model-risk teams want to know exactly what data trained the model, where it came from, and whether the institution has rights to use it. This matters legally—if training data includes proprietary customer information from other institutions or copyrighted materials, the bank faces intellectual property and privacy risks.

The question intensifies when banks fine-tune foundation models on internal data. MUFG's implementation of ChatGPT Enterprise, as documented by OpenAI, shows one approach: the bank uses the platform to 'build an AI-native organization' while maintaining control over proprietary data. But model-risk teams will still ask: does fine-tuning on customer transaction data create privacy obligations? If the model inadvertently memorizes PII, can it be compelled to 'forget' specific records under data protection laws?

Operators should document: the complete data lineage for any fine-tuning, contracts with model providers specifying data retention and usage policies, and technical controls preventing the model from outputting verbatim training data. Concrete answers matter more than general assurances. 'We use GPT-4' doesn't address whether customer data was used in continued training or whether the vendor's terms prohibit certain financial applications.

Traditional models get validated before deployment and monitored continuously. A credit model might be backtested against historical default data, then tracked monthly for prediction accuracy. Generative AI breaks this workflow. The model updates frequently, sometimes without explicit version changes. Prompt engineering can alter outputs dramatically without touching model weights. And unlike a binary classifier, 'correct' output for a generative model is often subjective.

Australian Payments Plus, in their documented ChatGPT Enterprise use case, addresses this by keeping 'human judgment central' even while using AI to accelerate analysis. That's the pragmatic answer model-risk teams accept: not that the model is perfectly accurate, but that human review catches errors before they reach customers or regulators.

Effective monitoring plans include: baseline accuracy testing on representative tasks with known correct answers, human review of a statistical sample of outputs before customer delivery, red-team testing with adversarial prompts designed to elicit harmful or incorrect responses, and version control that documents when prompts, models, or retrieval systems change. Model-risk teams will ask to see logs proving this monitoring actually happens, not just policy documents saying it should.

Banks have entire departments managing third-party vendor risk. When you deploy a generative AI system using a commercial API, you've introduced a critical dependency on a vendor that can change model behavior, pricing, or availability without your input. Model-risk teams have seen this movie before with every SaaS platform and core banking system.

The questions are specific: What's your fallback if the API goes down during market hours? If the vendor changes the model and accuracy degrades, how quickly can you detect and respond? What contractual commitments does the vendor make about model stability, and what remedies do you have if they break those commitments? OpenAI's recent government and national security partnerships announcement emphasizes 'democratic accountability' and 'responsible AI use,' but those principles don't automatically translate to contractual service-level agreements.

Operators should prepare: documented fallback procedures, performance benchmarks that trigger alerts if accuracy drops, contract terms specifying notification periods before major model changes, and an exit strategy. The exit strategy is the one most operators skip. If you've built critical workflows around GPT-4, what's your plan if pricing triples or the vendor exits the market? Model-risk teams won't approve deployment without an answer.

Fair lending laws and anti-discrimination regulations apply to AI systems just as they do to human loan officers. Model-risk teams will ask whether your generative AI system produces disparate outcomes based on protected characteristics like race, gender, or age—even if those characteristics aren't explicit inputs.

This isn't hypothetical concern. Generative models trained on internet text absorb societal biases present in that text. A model asked to 'write a professional bio' might default to male pronouns for executives or suggest different investment products based on names associated with specific ethnicities. The Google deepfake detector case recently made news for debunking a fake image of a political figure—but that same technology highlights how easily AI systems can be manipulated or produce unreliable outputs when edge cases appear.

Practical mitigation includes: testing outputs across demographic groups to detect disparate impact, implementing business rules that remove protected characteristics from prompts, maintaining audit logs that allow you to investigate complaints about biased outcomes, and training human reviewers to recognize bias patterns. Model-risk teams expect statistical evidence, not assurances. Show them testing results comparing outcomes across customer segments, using real or representative data.

The common thread through all these questions is documentation. Model-risk teams operate in a world where 'we think it works well' gets replaced with 'here's the validation report, monitoring dashboard, and independent review.' For operators, this means treating generative AI deployment like any other model deployment: formal validation before production, ongoing monitoring with defined thresholds, and change management processes when prompts or models update.

The MIT Technology Review's recent piece on foundational AI architecture notes that organizations expanding into agentic systems must account for constant evolution while managing risk—the exact tension model-risk teams navigate. The practical approach is staged deployment: start with low-risk use cases where errors don't harm customers or violate regulations, build monitoring and documentation practices, then expand to higher-risk applications once you've proven you can manage the model effectively.

Smart operators come to model-risk reviews with evidence, not promises. Bring validation test results, monitoring dashboards showing real production data, documentation of your change control process, and specific answers to the questions outlined here. Model-risk teams aren't there to block innovation—they're there to ensure the institution can manage the risk you're introducing. Meet them with operational rigor, and they become allies in scaling AI across the organization.