Why Your Pentest Vendor Isn't Testing for AI Vulnerabilities

Your organization probably runs penetration tests on a regular cycle. Quarterly, annually, maybe on every major release. You get a report, you remediate the findings, and you file it for your next audit. The process works well for traditional web applications, network infrastructure, and cloud configurations.

But if you have deployed AI-powered features — a chatbot that handles customer inquiries, a recommendation engine, an internal tool that uses LLMs to process documents — your pentest almost certainly did not touch them. Not because your vendor is negligent, but because AI security is a fundamentally different discipline that most security testing firms are not equipped to assess.

What Makes AI Security Different

Traditional application security focuses on well-understood vulnerability classes: injection attacks, broken authentication, misconfigurations, insecure deserialization. The OWASP Top 10 has been a reliable framework for decades. Testers know what to look for, and automated tools can detect many of these issues.

AI systems introduce entirely new attack surfaces that do not map to traditional vulnerability taxonomies.

Prompt injection is the SQL injection of the AI era. Attackers craft inputs that cause an LLM to ignore its instructions, bypass safety guardrails, or execute unintended actions. Direct prompt injection targets the model through user input. Indirect prompt injection embeds malicious instructions in data the model processes — a website the model summarizes, a document it analyzes, an email it reads. This is not theoretical. Researchers have demonstrated prompt injection attacks that exfiltrate user data, bypass access controls, and manipulate model outputs in production systems.

Training data poisoning targets the data used to train or fine-tune models. If an attacker can influence training data, they can embed backdoors that cause the model to behave maliciously under specific conditions while appearing normal otherwise. This is particularly relevant for organizations that fine-tune models on their own data or use retrieval-augmented generation (RAG) with data stores that could be manipulated.

Model extraction involves querying a model systematically to reconstruct its behavior, effectively stealing the model or its training data through inference. For companies that have invested significantly in proprietary models, this represents direct IP theft through an attack vector that traditional pentests never evaluate.

Jailbreaking bypasses a model's safety alignment to produce outputs it was designed to refuse — harmful content, instructions for dangerous activities, or responses that violate compliance requirements. If your customer-facing AI can be jailbroken, your brand is exposed.

Why Traditional Vendors Miss It

This is not an indictment of your pentest provider. Most security testing firms built their practices around web, network, and application security. That is what they are good at, and those assessments remain important.

The problem is threefold. First, scope. Standard pentest engagements are typically scoped around network ranges, web applications, and API endpoints. AI components are either explicitly excluded or treated as black boxes that receive only surface-level testing — the tester verifies the API endpoint requires authentication but does not test whether the model itself can be manipulated through its intended input channels.

Second, expertise. Testing AI systems requires understanding how language models process inputs, how RAG architectures retrieve context, how agent frameworks chain model calls, and where the trust boundaries exist in ML pipelines. This is specialized knowledge that most security consultants have not yet developed.

Third, tooling. The automated scanning tools that form the backbone of traditional pentests — Burp Suite, Nessus, Metasploit — were not designed to test for prompt injection or evaluate model robustness. AI security testing requires specialized tooling and manual techniques that most firms do not have in their arsenal.

Real Examples of AI-Specific Vulnerabilities

In one engagement, we found that a customer service chatbot could be prompt-injected to return internal system prompts that contained database connection patterns and API key formats. The chatbot passed its traditional security review because the API was properly authenticated and encrypted. Nobody thought to ask what happened when a user typed creative instructions instead of a normal question.

In another case, a document analysis tool using RAG was pulling from a shared knowledge base that included user-contributed content. By adding a carefully crafted document to the knowledge base, we were able to manipulate the model's responses to other users' queries — a classic indirect prompt injection that turned a shared resource into an attack vector.

We have also seen AI systems that leak training data through their outputs. One model, when prompted in specific ways, would reproduce verbatim passages from confidential documents it had been fine-tuned on. The application's access controls were sound, but the model itself had memorized sensitive data and would surface it to any authenticated user who knew how to ask.

What a Proper AI Security Assessment Covers

A comprehensive AI security assessment goes well beyond what traditional pentests address. It should include an architecture review of how your AI systems are designed, how data flows through them, where trust boundaries exist, and how model outputs are used in downstream decisions.

It needs thorough prompt security testing — systematic attempts to bypass guardrails, extract system prompts, manipulate outputs, and escalate privileges through both direct and indirect injection techniques.

A data pipeline review examines how training data is sourced, validated, and protected; how RAG retrieval stores are populated and secured; and whether data poisoning is a realistic attack path.

And integration security testing evaluates how the AI system connects to your broader infrastructure — what APIs it can call, what data it can access, whether output sanitization prevents the model from triggering actions it should not be able to trigger.

The result is a clear picture of your AI-specific risk exposure with prioritized, actionable findings — not a generic report, but a map of the attack paths that matter for your specific architecture.