Guest
Watch Full Episode
Introduction
Jay Combs: Hello, and welcome to the ModelOp Good Decisions podcast. I'm Jay Combs, the VP of Marketing. Today, we're going to be talking about ModelOp's new AI Governance Score—what it is and why it's important. Joining us is our VP of Product, Dave Trier. Dave, welcome to the podcast. How are you?
Dave Trier: Doing well. Thanks for having me, Jay. Looking forward to this.
Jay Combs: Awesome. Before we get into the technicalities and concepts of the AI Governance Score, can you give us a quick introduction about yourself, your journey to ModelOp, and your focus on AI governance?
Dave Trier: Absolutely. My background is in working with enterprises to deliver data analytics, data science, ML, and AI. During this time, I often encountered major bottlenecks in using these analytics for production business purposes, primarily due to operational and governance concerns. This highlighted a need in the market for proper oversight, governance, and operational procedures in AI and ML. Over the past five to seven years, I have focused on AI and ML governance to address these needs.
Understanding the AI Governance Score
Jay Combs: Got it. That’s a good summary of why the AI Governance Score is important. We know there's a larger AI accountability challenge. Consulting firms like Accenture state that only about 2% of executives or CEOs have insight into how AI is being used, what it's doing, and the associated risks within a company. Let's dive into the details. What is an AI Governance Score, and why is it important?
Dave Trier: Absolutely. As enterprises adopt various technologies like machine learning, traditional statistics, neural nets, deep learning, and generative AI, it becomes challenging for governance officers and executives to ensure these models adhere to governance policies. The AI Governance Score provides a consistent metric to measure adherence across all AI and ML initiatives, regardless of the technology or source. It allows for an apples-to-apples comparison to understand if all models and capabilities comply with governance policies.
Components of the AI Governance Score
Jay Combs: Being a metric, I imagine there are many factors that roll into it, which might differ by industry or company. Can you elaborate on what goes into this metric?
Dave Trier: Certainly. The AI Governance Score is configurable to your specific governance policy. It includes various factors:
- Information Collection: Gathering required business, technical, and metadata information about your AI use case.
- Asset Documentation: Documenting assets related to the AI use case, such as source code, binary artifacts, execution details, and configurations of vendor models.
- Evidence Provision: Providing necessary evidence like test results, documentation, approvals, and reports.
- Control Adherence: Ensuring controls like attestations, change controls, process controls, and data controls are in place and followed.
These categories help provide a comprehensive AI Governance Score.
Importance of the AI Governance Score
Jay Combs: That makes a lot of sense. The AI Governance Score offers quick insights for executives who may not be involved in the day-to-day details of the AI model lifecycle. It helps them understand where everything stands and what the risks are.
Dave Trier: Exactly. It offers consistency across different technologies and provides visibility for executives who need to ensure the enterprise is protected and policies are followed. It also helps enforce governance policies beyond just word-of-mouth and training, providing a structured way to ensure adherence.
Jay Combs: How did you come up with the idea for the AI Governance Score? Was it through conversations with customers or market research?
Dave Trier: We've had the pleasure of speaking with hundreds of executives who have invested heavily in AI. They often expressed a need for visibility into where AI is being used and the associated risks. This feedback drove the development of the AI Governance Score, providing executives with the necessary visibility and risk assessment across different AI technologies.
Implementation of the AI Governance Score
Jay Combs: Conceptually, this sounds great. How do you actually implement it? Enterprise tech implementations can be challenging. How do you get off the ground with this?
Dave Trier: ModelOp has invested heavily in being technology-agnostic, integrating seamlessly with various technologies, and automating the process. This includes collecting all necessary model information, integrating with different tech stacks, and automating adherence to governance policies. Our software is designed to streamline this process, making it less manual and more efficient.
Getting Started with ModelOp
Jay Combs: If I were a Chief AI Officer or CIO wanting to use ModelOp and get an AI Governance Score, what steps would I need to take? How long does it take to set up?
Dave Trier: Great question. ModelOp Center comes with pre-configured templates based on general regulations like NIST AIRMF and OCC's SR 11-7. These templates cover about 70-80% of what you need, and you can customize them to your enterprise's specifics. This allows you to get up to speed quickly.
Final Thoughts
Jay Combs: So, the AI Governance Score is a standardized metric for measuring compliance and adherence, providing visibility and accountability across the enterprise. Anything else you want our listeners to know about the AI Governance Score?
Dave Trier: Just that it’s incredibly powerful for providing visibility at all levels, even to the board of directors. It’s a consistent way to ensure that all AI systems are adhering to governance policies. For those in charge of AI governance, I strongly recommend looking into how ModelOp can help enforce and provide visibility into your AI governance.
Jay Combs: Thank you, Dave. This was a great episode. Thanks for sharing your thoughts on the AI Governance Score. That’s it for this episode of Good Decisions. Thanks for joining us. Please subscribe to the podcast, and if you have ideas or questions, reach out to us. We'd love to connect and discuss more about AI governance. Thanks so much. Bye.
