1. How is our data used, and is it used to train your models?
This is the most critical question. You need to know exactly what data an AI tool collects, where it goes, and its purpose. The key concern is whether your proprietary business information or customer data will be used to train the vendor's large language
model (LLM), potentially exposing it to other customers. A trustworthy vendor will provide a clear, legally binding policy stating that your data is never used for training their general models and that your instance is logically or physically separate.
2. How do you protect the AI model from being attacked?
AI models themselves are a new and valuable attack surface. Ask vendors how they defend against threats like data poisoning, where an attacker corrupts the training data to manipulate the model's future outputs. Also, inquire about model inversion, where attackers can sometimes reverse-engineer a model to extract the sensitive data it was trained on. A secure AI provider should have robust defenses to ensure the integrity and confidentiality of their models throughout the entire lifecycle.
3. What measures are in place to prevent prompt injection and insecure outputs?
Prompt injection is a major vulnerability where an attacker uses crafty inputs to trick the AI into ignoring its original instructions and performing malicious tasks. This could lead to data leaks or the generation of harmful content. Equally important is how the AI's output is handled. An application that blindly trusts AI-generated content can be exploited. Inquire about the input validation and output filtering mechanisms the vendor uses to mitigate these OWASP Top 10 LLM risks.
4. What is your process for securing the AI supply chain?
Modern AI systems are not built in a vacuum. They often rely on third-party models, open-source libraries, and various datasets. This complex supply chain introduces risk. A single compromised component could create a vulnerability in the entire system. Ask vendors how they vet, monitor, and patch third-party components and data sources to ensure they haven't been tampered with or compromised. This demonstrates their commitment to end-to-end security.
5. How do you manage access controls for the AI and its data?
Not everyone in your organization needs the same level of access to a powerful AI tool. Ask potential vendors about their access control features. A secure system should enforce the principle of least privilege, ensuring users and even the AI agents themselves only have access to the data and functions necessary for their roles. Strong identity management, multi-factor authentication, and detailed audit logs are non-negotiable for any enterprise-grade AI solution.
6. How do you ensure the AI complies with privacy regulations?
The regulatory landscape for AI is evolving rapidly, with frameworks like the EU AI Act joining established laws like GDPR and CCPA. It is essential that any AI vendor can articulate how their solution meets these compliance obligations, especially regarding data residency, data minimization, and user rights like data deletion. Ask for their compliance certifications and third-party audit reports, which provide objective proof of their security posture.
7. Can we audit the AI's actions and understand its reasoning?
Trusting a "black box" is not a viable security strategy. For accountability and incident investigation, you need transparency. Ask if the AI system provides clear, human-readable logs of its actions and, where possible, explanations for its decisions. This concept, known as explainability, is crucial for debugging, auditing, and building trust with both users and regulators. Without it, determining the root cause of an AI-driven security incident becomes nearly impossible.
8. What is your incident response plan for an AI-related breach?
Even with strong defenses, you must plan for the worst. Ask vendors about their specific incident response plan for AI-related security events. This includes how they would detect a breach, what their notification process is, how they would contain the damage, and how they would work with your team to remediate the issue. A vague answer is a major red flag; a mature provider will have a well-documented and tested process.
9. How does the AI handle sensitive information disclosure?
One of the most significant risks of generative AI is its potential to inadvertently leak sensitive information it has access to. This can happen if an employee pastes proprietary code into a public chatbot or if a model accidentally includes personally identifiable information (PII) in its response. Ask vendors what specific safeguards, such as data sanitization and output filtering, they use to prevent the model from revealing confidential business data, employee details, or customer PII.
10. What training and governance support do you provide?
A secure tool is only effective if people use it correctly. The risk of "shadow AI"—employees using unvetted public tools—is a major threat. A good AI partner should provide resources to train your employees on safe usage. Ask what documentation, training materials, and governance frameworks they offer to help you build an internal policy for responsible AI use. This fosters a culture of security and ensures the technology is an asset, not a liability.
