Cybersecurity researchers have revealed a severe security flaw in the Vanna.AI library that can be exploited to achieve remote code execution through prompt injection techniques. The vulnerability, identified as CVE-2024-5565 (CVSS score: 8.1), involves a prompt injection issue in the “ask” function, which can be exploited to trick the library into executing arbitrary commands, according to supply chain security firm JFrog. Vanna is a Python-based machine learning library that enables users to interact with their SQL database by asking questions, which are translated into SQL queries using a large language model (LLM).
The rapid adoption of generative artificial intelligence (AI) models in recent years has highlighted the risks of exploitation by malicious actors who can use adversarial inputs to bypass built-in safety mechanisms. One prominent type of attack is prompt injection, which involves bypassing the guardrails set by LLM providers to prevent the production of offensive, harmful, or illegal content, or to execute unintended instructions. Such attacks can be indirect, where a system processes third-party controlled data (e.g., incoming emails or editable documents) to launch a malicious payload that leads to an AI jailbreak.
They can also manifest as a many-shot jailbreak or multi-turn jailbreak (aka Crescendo), where the attacker begins with harmless dialogue and gradually steers the conversation toward the prohibited objective. This method can be extended to perform another novel jailbreak attack known as Skeleton Key. “This AI jailbreak technique uses a multi-turn strategy to make a model ignore its guardrails,” explained Mark Russinovich, chief technology officer of Microsoft Azure. “Once the guardrails are bypassed, the model cannot differentiate between malicious and legitimate requests“.
Skeleton Key differs from Crescendo in that, once successful, the model’s system rules are altered, allowing it to generate responses to otherwise forbidden questions, regardless of ethical and safety risks. “When the Skeleton Key jailbreak is successful, a model acknowledges that it has updated its guidelines and will comply with instructions to produce any content, no matter how much it violates its original responsible AI guidelines,” Russinovich added. Unlike other jailbreaks like Crescendo, where tasks must be requested indirectly or with encodings, Skeleton Key allows direct task requests, and the model’s output appears unfiltered, revealing the full extent of its knowledge or ability.
The latest findings from JFrog, also independently disclosed by Tong Liu, demonstrate how prompt injections can have severe impacts, especially when linked to command execution. CVE-2024-5565 exploits Vanna’s text-to-SQL generation feature, which creates SQL queries and graphically presents them to users using the Plotly graphing library. This is done through an “ask” function, such as vn.ask(“What are the top 10 customers by sales?”), which generates SQL queries to be run on the database. The combination of this behavior with the dynamic generation of Plotly code creates a security vulnerability, allowing a threat actor to submit a specially crafted prompt that includes a command to be executed on the underlying system.
“The Vanna library’s prompt function can be altered using prompt injection to run arbitrary Python code instead of the intended visualization code,” JFrog stated.n “Allowing external input to the library’s ‘ask’ method with ‘visualize’ set to True (default behavior) leads to remote code execution.” Following responsible disclosure, Vanna issued a hardening guide warning users that the Plotly integration could generate arbitrary Python code and advised using this function in a sandboxed environment.
“This discovery highlights the significant risks of widespread use of GenAI/LLMs without proper governance and security,” said Shachar Menashe, senior director of security research at JFrog. “Prompt injection dangers are not widely known but are easy to execute. Companies should not rely solely on pre-prompting as a defense and should implement more robust mechanisms when interfacing LLMs with critical resources such as databases or dynamic code generation“.
- Advertisement -
, involves a prompt injection issue in the "ask" function, which can be exploited to trick the library into executing arbitrary commands, according to supply chain security firm JFrog. Vanna is a Python-based machine learning library that enables users to interact with their SQL database by asking questions, which are translated into SQL queries using a large language model (LLM).){kind=link}