As the financial industry increasingly integrates AI into its core operations, it’s never been more critical to have rigorous model governance capabilities.
In this webinar, Sumalatha Bachu, Senior Director, Technology, and Harvey Westbrook, Senior Director, Regulatory Economics & Market Analysis at FINRA share their insights on model governance and analytics, business and technical challenges, and how they impact FINRA's priorities and mission.
Webinar Highlights:
- Introduction to Model Operations: Understand the fundamental concepts and why they are crucial in today's tech-driven market environment.
- Implementing Model Governance: Learn how FINRA jump started their governance implementation for models and data science analytics workflows
- Expert Insights: Hear from FINRA’s Sumalatha Bachu and Harvey Westbrook, who bring a wealth of knowledge and firsthand experience in implementing and overseeing effective model governance and analytics.
Transcript
1. Introduction to AI Governance and FINRA
Jay Combs: Welcome to "Good Decisions," our monthly AI governance webinar. We're thrilled to have two esteemed guests with us today—senior directors from the Financial Industry Regulatory Authority, also known as FINRA. This webinar will be a little different from our past ones; we’re doing a fireside chat, where our guests will share insights on getting started with, implementing, and scaling model governance, managing change, and addressing both the business and technical challenges that come with it.
Let's jump right in. I'm just going to the next slide here for our agenda. FINRA is at the forefront of model operations, bringing together cross-functional teams to innovate and address organizational-level goals.
2. Introducing the Guests
Jay Combs: Today, we’re very fortunate to have two senior directors with us. One comes from the technical side, and the other from the business side, so we’ll get a well-rounded perspective on model operations.
Please welcome Sumalatha Bachu and Dr. Harvey Westbrook. Sumalatha is the Senior Director of Technology. She has been with FINRA for over twenty years, specializing in big data operations, data analytics, and model operations. She leads multidisciplinary teams that manage market regulation data operations at a massive scale.
Dr. Harvey Westbrook is Senior Director and Assistant Chief Economist, specializing in regulatory economics and market analysis. He’s been with FINRA for over three years, bringing deep experience from both the public and private sectors, leading analytics projects at every life cycle stage to improve business decision-making.
Welcome to both of you, and thanks for joining us today. Let’s dive right in—Harvey, I’d like to start with you.
3. Understanding FINRA's Role
Jay Combs: Many folks on the call might already be familiar with FINRA, but just to ensure we’re all on the same page—Harvey, could you give us an overview of FINRA and what you do?
Harvey Westbrook: Thanks, Jay. FINRA is a self-regulatory organization authorized by Congress to protect America’s investors by ensuring that the broker-dealer industry operates fairly and honestly. We work under the supervision of the Securities and Exchange Commission (SEC) and enforce rules governing ethical activities of all registered broker-dealer firms and brokers in the U.S. We also examine member firms for compliance, foster market transparency, and educate and protect investors.
To enable these regulatory responsibilities, we have vast and complex data resources that we manage to fulfill our mission of safeguarding the investing public against fraud and bad practices. My organization within FINRA is the Office of Regulatory Economics and Market Analysis (REMA), led by FINRA's Chief Economist, Jonathan Sokobin. REMA's responsibilities include conducting economic impact assessments of FINRA rulemakings, performing data analytics, supporting regulatory operations, and monitoring emerging trends in the financial sector, such as developments in blockchain and fintech.
Jay Combs: Wow, that's quite comprehensive. Thanks for breaking that down.
4. The Business Side of FINRA
Jay Combs: Sumalatha, could you tell us where you fit into the organization and your role at FINRA?
Sumalatha Bachu: Certainly. Thanks, Jay. Good afternoon, everyone. I’m excited to share my experiences alongside Harvey on model operations, model governance, and analytics.
As Harvey mentioned, we regulate and monitor the brokerage industry and the stock markets in the U.S. Our mission is to protect investors by ensuring fair and honest market operations, preventing market manipulation, and disciplining those who break the rules. To do this, we collect massive amounts of data—and when I say massive, I mean it. We ingest data from various exchanges and financial firms, processing over 620,000 brokers, regulating 3,300 securities firms, and handling 600 billion data transactions every day. We work with 24 different stock market exchanges in the U.S. and manage around 500 terabytes of data.
In my role, I oversee the big data processing ecosystem, managing data ingestion, extraction, transformation, quality, and all the complexities that come with handling such large volumes of data. The importance of good data cannot be overstated, as it forms the foundation for making good decisions and ensuring effective model governance.
Jay Combs: That’s a massive responsibility—truly impressive.
5. Importance of AI and Machine Learning
Jay Combs: Let’s dive into the model operations and governance side of things. Harvey, why are artificial intelligence, machine learning, and building models so critical to FINRA?
Harvey Westbrook: Great question, Jay. As Sumalatha pointed out, our data resources are enormous, complex, and extremely rich. FINRA’s regulatory mission can be thought of in five main ways: determining misconduct, enforcing rules, preventing wrongdoing, disciplining those who break the rules, educating investors, and resolving disputes. Given the scale and variety of data we have, analytics is essential to accomplish all of this effectively.
We utilize various analytical techniques and models to inform our regulatory work—from detecting and preventing misconduct to understanding compliance challenges at member firms. We use data to examine the impacts of regulatory changes and better understand who those changes will affect among market participants.
Jay Combs: Got it. Sumalatha, how does FINRA specifically use machine learning models to make decisions?
Sumalatha Bachu: Absolutely. It’s important to remember that, while decisions are ultimately made by people, those decisions are informed by machine learning model predictions. These models generate insights that help us make data-driven decisions, improving operational efficiency. At FINRA, we integrate machine learning models into our business processes, from data collection and preparation to model development, integration, deployment, and monitoring. It’s a continuous, iterative process that aims to enhance efficiency, accuracy, and responsiveness in meeting regulatory requirements.
Jay Combs: That makes a lot of sense—it’s impressive how data is truly at the core of everything you do.
6. Challenges in Model Operations
Jay Combs: Let’s talk about some of the challenges. Sumalatha, what are the biggest technical challenges that you face in model operations?
Sumalatha Bachu: Great question, Jay. The challenges we face can be categorized into three main areas: people, processes, and tooling. From a technical standpoint, one of the biggest challenges is dealing with the complexities of machine learning models—ensuring data security, managing vast data volumes, and utilizing the right technologies. Another significant challenge is establishing a common language for risk management across both business and technology teams. We need to make sure everyone understands each other, and proper training between teams is crucial to achieving this.
On the tooling side, FINRA has diverse business units with varying needs, which means we use different tools and technologies. However, we also need centralized governance to ensure consistency. We aim to have one central location for monitoring model governance, which is where our current efforts are focused.
7. Starting Model Operations and Governance
Jay Combs: That’s a lot to navigate. Harvey, what are some of the challenges you see from the business side, and how do you bring the teams together to solve these problems?
Harvey Westbrook: Absolutely, Jay. Building useful analytics requires collaboration across the organization. It’s a new way of working that involves teams with very different expertise coming together at different stages of a project. We need business experts, analysts, and technologists—all working cohesively. Often, people don’t have deep expertise in areas outside their own, which makes coordination and planning challenging.
One key element is establishing clear roles and responsibilities. Each team must understand not only their contribution but also the impact of their work on other parts of the project. Communication is vital. For instance, a business expert may understand the requirements, but we need someone who can translate that into an empirical approach for our analysts and technologists.
Technology evolves rapidly, and that can impact our work in ways that are difficult for everyone to understand. At FINRA, we have focused on building workflows that help teams understand these trade-offs. We’ve partnered with Georgetown University to provide staff with analytical training, which has helped bridge the gaps between different teams and encouraged the development of a shared language around analytics.
8. Key Takeaways on Model Governance
Jay Combs: It sounds like cultural alignment is key. So, Harvey, what are the main takeaways for our audience when it comes to model governance?
Harvey Westbrook: Start small and learn from your experiences. Analytical work is iterative—you learn about the questions you need to ask and the data you have at every stage of the project. It’s important to create an environment where teams can innovate while being accountable for delivery. Leaders need to foster trust and encourage collaboration, which allows teams to scale effectively.
Jay Combs: Sumalatha, what’s the most important thing you’ve learned from your experience in model governance?
Sumalatha Bachu: Governance doesn’t start at the end of the project; it starts at the very beginning. It’s a part of every stage of the model development lifecycle, from inception to deployment and beyond. Everything is constantly changing—technologies evolve, and we need to adapt. For me, one of the biggest lessons is the importance of minimizing manual interventions and focusing on continuous improvement. We have to build and operate in a way that maintains stability while embracing change.
Jay Combs: Those are excellent insights. Thank you both for sharing your perspectives.
9. Consequences of Ignoring Model Governance
Jay Combs: Let’s switch gears a bit. Harvey, what are the repercussions of not implementing model governance?
Harvey Westbrook: Great question, Jay. Models inherently have uncertainty, and it’s crucial for organizations to understand the different types of uncertainty present in their data and models. When governance is absent, there is a lack of accountability, which increases the risks associated with inaccurate model predictions or unintended biases. Without proper governance, organizations may not effectively manage these uncertainties, leading to reputational damage, regulatory non-compliance, and financial losses.
It’s really about ensuring shared responsibility and governance, as well as organizational trust. This trust is built when teams understand the trade-offs and risks associated with each model, and governance helps establish a framework for managing these risks.
Jay Combs: That makes sense, Harvey. Sumalatha, what about from a technical perspective—what happens if governance isn’t prioritized?
Sumalatha Bachu: From the technical side, governance needs to start right at the beginning. It’s not something to think about after deployment. If we don’t integrate governance into each stage of the model life cycle—whether it’s data collection, model testing, or monitoring—it can lead to inefficient processes and even complete failure of the model. With changes happening so quickly in technology, governance helps us ensure that models are compliant, secure, and maintain their intended performance over time.
10. Initiating Governance: First Steps and Best Practices
Jay Combs: So, Sumalatha, how did you initiate model governance at FINRA? What were the first steps?
Sumalatha Bachu: It was definitely a journey, Jay, and it didn’t happen overnight. The first thing we did was establish the processes. We identified the tools we needed and recognized the cultural shift required to make model operations and governance a reality.
We started with strategy and planning. We defined our objectives, identified the necessary infrastructure, and determined the automation needs for each stage of the model life cycle. Once those aspects were established, we performed R&D, conducted a proof of concept (POC), and then piloted our approach.
Starting small was key for us. We learned a lot in the initial phases, and then we scaled up. We also adopted a risk-based model life cycle, which allowed us to govern machine learning models while keeping them ethical, transparent, compliant, and secure.
Jay Combs: And, Harvey, how did you work to bring the business and technical teams together on this journey?
Harvey Westbrook: For us, culture was a huge factor. We needed a culture that embraced analytics and encouraged teams to learn a common analytical language. We partnered with Georgetown to provide training across our teams, which allowed people from different areas of expertise to come together and understand the fundamentals of analytics and governance.
This common understanding was critical for building workflows that effectively integrated governance at each stage. It allowed us to align the interests of both business and technical teams, creating a shared sense of responsibility.
11. Future Priorities: Scaling and Sustaining Model Operations
Jay Combs: So what are your priorities for the next year, Sumalatha, when it comes to model operations and governance?
Sumalatha Bachu: We’re focusing on scaling and sustaining model operations. This includes establishing standard operating procedures, both for traditional models and for Generative AI. It’s important to have effective governance across both types while balancing innovation and risk.
We are also focusing on post-production model maintenance—detecting data drifts, managing performance issues, and minimizing the time required for course corrections. As Harvey mentioned, Gen AI presents tremendous opportunities, but it also brings new risks. Effective governance will be key in managing both.
12. Strategic Use Case Selection for Governance
Jay Combs: A question from the audience—how do you assess use cases to pursue in model governance?
Harvey Westbrook: That’s a great question. There are strategic considerations that come into play. First, you need to understand what is most valuable for your organization—what presents the greatest benefit? At the same time, you need to think about your risk appetite. Some use cases may have more risk, but also higher rewards. It’s a balancing act to determine where to start and what resources to allocate.
13. Key Takeaways: Starting Governance Initiatives
Jay Combs: We’re almost at time, so let’s recap the key takeaways from today’s discussion. Sumalatha, Harvey, what should people remember about starting model governance?
Harvey Westbrook: Don’t wait—start now. Model governance is iterative, and you’ll learn as you go. It’s all about building trust and encouraging collaboration across teams. If you haven’t started, you’re already behind.
Sumalatha Bachu: I agree. Governance has to be in place from the very beginning, not just as an afterthought. Alignment among key stakeholders is crucial, as is having clear documentation and processes in place. You need to work through challenges iteratively, learn from your experiences, and keep improving.
14. Upcoming AI Governance Leadership Summit Announcement
Jay Combs: Before we wrap up, I have an exciting announcement. We’re hosting the 2024 AI Governance Leadership Summit on September 25th. Last year’s summit was a huge success, bringing together members of the AI community, innovation teams, technical teams, and security teams to share insights and exchange ideas.
We’ll have keynote speakers, another fireside chat, and a panel of experts. Registration is now open, and we’ll be announcing the speakers and agenda in the coming weeks. We’d love to see you there—it's going to be a great event.
15. Conclusion and Invitation for Further Discussion
Jay Combs: And with that, we conclude our webinar. Thank you so much to both of you—Sumalatha and Harvey—for joining us and sharing your insights today. Your expertise is invaluable.
For those watching, if you have more questions or want to learn about model governance, feel free to reach out to us at ModelOp anytime. We’d love to continue the conversation. Thanks, everyone, and have a great rest of your week.
Harvey Westbrook: Thank you for having us, Jay.
Sumalatha Bachu: Thank you, everyone. Goodbye.
Jay Combs: Goodbye, everyone.