In May 2024, JFrog’s security research team disclosed CVE-2024-5565 in Vanna.AI, an open source Python library that turns a natural language question into a SQL query. The library also asked the model to write the Python charting code for the answer, and then ran that code. So a question could become a program. Nothing about the model was broken. The security decision sat one layer below it, where generated text was handed to an execution call with nothing in between. Any product where model output reaches an interpreter, a shell, a query engine, a template, or a browser has the same shape, whatever the model is.
Key findings
- The record is CVE-2024-5565, published 31 May 2024, classified as CWE-94, Improper Control of Generation of Code, with a CVSS 3.1 base score of 8.1 assigned by JFrog as the CVE Numbering Authority. NVD had not published its own assessment at the time of checking.
- The entry point is prompt injection. The finding is output handling. In OWASP Top 10 for LLM Applications terms, LLM01 gets in and LLM05 Improper Output Handling is what makes it matter.
- The dangerous line is the sink, not the prompt. Generated code was executed. Remove or isolate that step and the same injection produces a bad chart instead of a bad outcome.
- The mitigation named in the advisory is to disable the code generation path when the input comes from outside your trust boundary, rather than to filter the input.
- The maintainer published a hardening guide in response, which is the pattern worth copying: document the trust boundary the library expects the integrator to hold.
- Text to SQL products usually carry two sinks. The generated query is the one teams defend. The generated chart, template, or export step is the one that gets missed.
This post is written by Rathnakara GN, who leads AI and LLM penetration testing at Cybersecify. It analyses a publicly disclosed issue from the published record. We did not test Vanna.AI, and nothing here describes a live weakness or contains reproduction detail. The technique is kept at the level a defender needs in order to recognise the same shape in their own product.
What was disclosed
Vanna.AI is a Python library that answers questions about a database in natural language. It uses retrieval to assemble context about the schema, prompts a model to write SQL, runs the query, and then returns a result. To make the result readable it also produced a chart, and the charting code itself was generated by the model rather than written once by a developer.
The published mechanism is a short chain. The user’s question goes into the library’s ask method. The model writes SQL. The chart generation step then builds a second prompt that carries the original question and the generated SQL into it, and the Python it produces is passed to an execution call. The NVD entry states the consequence plainly: external input to the ask method with visualisation enabled leads to remote code execution.
The advisory’s mitigation is to turn the visualisation argument off when the input is untrusted, and the maintainer added a hardening guide for integrators.
That is all the intrusion detail this post needs. The rest is the part you can act on.
The part that matters: the sink, not the model
There is a reflex in most engineering teams to read an incident like this as a model problem, then to respond with a better model, a stricter system prompt, or an injection classifier in front of the input.
That reflex misplaces the boundary. A model produces text. It has no way to know whether the text it just produced will be shown to a human, stored in a row, or executed on a server. That judgement is made by the application, and the application made it here by calling an execution function on a string whose content was influenced by a user.
Read the same feature with the model removed and it becomes an old and familiar bug. Untrusted input reached a code interpreter. We have had a name and a fix for that class for twenty years. The model did not create a new vulnerability class, it created a new and much less obvious path into an existing one, because the string that reaches the interpreter now travels through a component that looks like a language feature rather than like data flow.
This is why the useful question after any AI incident is not what the model was tricked into saying. It is where the thing it said went next.
What to test in your own product
None of the following requires knowing anything about Vanna’s implementation. Work through it in order.
1. Draw the line where generated text stops being text
Pick one AI feature and follow a single response from the moment the model returns it to the moment the user sees it. Write down every function it passes through. Somewhere on that path the response either stays a string that gets rendered, or it becomes an instruction that something else acts on. That transition point is the only place in the feature where a security decision is being made, and in most codebases nobody has explicitly decided it.
2. Inventory every sink
Once you know what a sink looks like in your product, list them all. The recurring ones:
- an evaluation or execution call in Python, JavaScript, or Ruby
- a shell or subprocess invocation
- a database query, including one assembled by an ORM
- a template renderer, especially one with expression support
- a markdown or HTML surface that can run script or load remote images
- a file path, an archive extraction, or an upload destination
- an HTTP client or webhook whose destination the model chose
- any tool or function call the model may invoke with arguments it wrote
Chart and dashboard generation belongs on that list, and it is the entry most often absent, because it is filed mentally under presentation. Anything an engineer describes as letting the assistant write a quick script belongs there too.
A fast first pass is to search the codebase for the execution primitives in your language, then trace backwards from each hit to see whether model output can reach it. The list is usually longer than the team expects, and it grows quietly, because adding a generated artefact feels like a product improvement rather than a security change.
3. Prefer a fixed set of operations over generated code
If a feature needs to produce a chart, the chart types your product supports are a finite list. Generating fresh code each time buys flexibility that most products never use, and pays for it with an execution path that carries user influence.
The safer construction is to have the model choose from a set of operations you wrote, and to supply parameters that you validate, rather than to have it author the operation. That converts an open ended code generation problem into a closed one where the failure mode is a wrong chart rather than a wrong process.
The same reasoning applies to queries, filters, exports, and workflow steps. Let the model pick and fill in. Do not let it author what runs.
4. Assume injection succeeds, then bound what it reaches
Prompt injection cannot be reliably filtered out with current models, so any defence built on recognising a malicious instruction has a shelf life. Design as though the instruction eventually lands.
- Run any generated artefact in an isolated environment with no network egress, a timeout, a memory ceiling, and no access to the host filesystem or environment variables.
- Give the database user the narrowest privilege the feature actually needs. Read only is a start. Read only against the specific tables the feature reads is better. Scoped to the requesting identity, where the data model allows it, is better still.
- Keep credentials and API keys out of the process that runs the generated artefact.
- Require human confirmation for anything with a side effect: a write, a payment, an email, a permission change.
Each of these turns a full compromise into a contained failure. None of them depends on getting the prompt right.
5. Log the artefact, not just the answer
Most teams log the request and the final response. Almost nobody logs the intermediate thing the model produced. That intermediate artefact is the only record that shows what actually executed, and without it an incident review can establish that a user asked a strange question and cannot establish what the system then did about it.
Log the generated query, the generated code, the tool call and its arguments, and the identity that triggered it. Retain them long enough to be useful during an investigation. This is also the difference between finding out from your own telemetry and finding out from a customer.
Where this sits in the frameworks
Two categories, in sequence:
| Stage | Category | Why |
|---|---|---|
| Entry | LLM01 Prompt Injection | Instructions arrive inside content the model treats as part of its task |
| Finding | LLM05 Improper Output Handling | Model output is passed into an execution context without validation or isolation |
| Underlying weakness | CWE-94 Improper Control of Generation of Code | The classic name for the bug once the model is removed from the diagram |
Reports that file this only under LLM01 tend to recommend prompt hardening, which does not close it. Filing it under LLM05 produces a remediation list you can finish. For the injection category in depth, see prompt injection 2026 attack patterns. For an incident where the same entry point led to a data exfiltration route instead of code execution, see our deep-dive on the Slack AI prompt injection disclosure.
Why pre-prompting is not a control
The JFrog write-up says it directly: developers should not rely on pre-prompting as an infallible defence.
The reason is structural. A system prompt is an instruction given to a component whose entire purpose is to follow instructions. It sits in the same context window as everything else, competing with the retrieved documents, the user’s question, and whatever else got assembled into the request. It is a strong preference, not a boundary.
Compare that with a sandbox with no network access, or a database role that cannot read the table in question. Those do not negotiate. When you are deciding where to spend a sprint on an AI feature, that is the distinction to spend it on.
What this means for an Indian SaaS company
The frameworks are jurisdiction neutral. The consequences are not.
If execution triggered through an AI feature leads to personal data reaching someone not authorised to see it, that is a personal data breach under the DPDP Act, and the disclosure path being a model rather than a database changes nothing about the obligation. Our DPDP breach response playbook covers the notification path.
The harder obligation is timing. CERT-In requires specified incidents to be reported within 6 hours of being noticed. For AI features the constraint is almost never the reporting, it is the noticing, and the gap closes at step 5 above rather than in the incident response plan.
What depth of testing finds this
Being specific about what different levels of assessment surface, because the difference is real work rather than a packaging exercise:
| Depth | What it surfaces here |
|---|---|
| Category coverage | The individual weaknesses. An injection surface on the input. A code execution call reachable from generated text. Both found, both real, both reported separately. |
| Systematic verification | Whether the boundary holds when exercised, rather than whether it exists on paper. This is where a sandbox that was configured but never had egress blocked gets separated from one that does. |
| Adversary emulation | The route. Input shaped at the generation step, producing an artefact that executes with the privileges of the service, reaching data the asking user was never entitled to. |
Category coverage finds findings in isolation. Adversary emulation finds the path between them. Real incidents are almost always paths.
We do not claim that testing prevents any specific incident. The narrower and more useful statement is this: tracing model output to every sink it can reach, then checking whether crafted input changes the artefact that runs rather than the answer that returns, is the test that surfaces this class. If your product generates code, queries, charts, or templates from something a user typed, that trace is the test worth asking any vendor to perform, including us.
How Cybersecify tests this class
Cybersecify is a founder-led penetration testing firm based in Bengaluru serving AI-first and API-first SaaS startups. Rathnakara GN, Co-founder and CHO, holds OSCP and leads AI and LLM engagements.
Our AI application pentest maps the feature end to end: retrieval sources, the tool graph, the privilege the service holds, and every point where generated text is interpreted rather than displayed. Findings map to the OWASP Top 10 for LLM Applications 2025 codes and to CWE identifiers so an engineering lead or an auditor can cross reference each issue. Where the AI feature sits on top of an API, the API pentest covers the layer underneath it.
For the framework itself, see our OWASP Top 10 for LLM Applications reference, and for agent-specific extensions, AI agent pentest methodology.
Sources
- NVD entry for CVE-2024-5565 (description, CWE-94, CVSS 3.1 base score 8.1 assigned by JFrog as CNA, published 31 May 2024). Checked 15 August 2026.
- JFrog security research advisory, Vanna prompt injection RCE (disclosing researcher’s advisory, mitigation guidance). Checked 15 August 2026.
- JFrog research write-up on the Vanna.AI issue (mechanism at the design level, vendor response, hardening guide). Checked 15 August 2026.
- OWASP GenAI Security Project, Top 10 for LLM Applications 2025.
Where to go from here
If your product generates code, SQL, templates, or charts from user input and you want to know whether that path is bounded, book a free 30-minute call with Ashok to scope the engagement, or see the pentest plans and pricing. For anything else, contact us.
We work with AI-first and API-first SaaS startups, Seed to Series B.