Getty Images/iStockphoto News How AI can attack corporate decision-making As AI gets embedded in corporate systems, experts warn of emerging security risks caused by influencing retrieval augmentation systems By Cliff Saran, Managing Editor Published: 30 Apr 2025 15:45 Given that the goal of developing a generative artificial intelligence (GenAI) model is to take human instructions and provide a helpful app, what happens if those human instructions are malicious? That was the question raised during a demonstration of AI vulnerabilities presented at the Centre for Emerging Technology and Security (CETaS) Showcase 2025 event in London. “A language model is designed to summarise large amounts of information,” said Matthew Sutton, solution architect at Advai. “The aim is to give it as much test information as possible and let it handle that data.” Sutton raised the question of what would happen if someone using a large language model (LLM) asked it to produce disinformation or harmful content, or reveal sensitive information. “What happens if you ask the model to produce malicious code, then go and execute it, or attempt to steal somebody’s data?” he said. During the demo, Sutton discussed the inherent risk of using retrieval augmented generation (RAG) that has access to a corpus of corporate data. The general idea behind using a RAG system is to provide context that is then combined with external inference from an AI model. “If you go to ChatGPT and ask it to summarise your emails, for example, it will have no idea what you’re talking about,” he said. “A RAG system takes external context as information, whether that be documents, external websites or your emails.” According to Sutton, an attacker could use the fact that the AI system reads email messages and documents stored internally to place malicious instructions in an email message, document or website. He said these instructions are then picked up by the AI model, which enables the harmful instruction to be executed.  “Large language models give you this ability to interact with things through natural language,” said Sutton. “It’s designed to be as easy as possible, and so from an adversary point of view, this means that it is easier and has a lower entry barrier to create logic instructions.” Read more about AI vulnerabilities NCSC warns over possible AI prompt injection attacks: The UK’s NCSC says it sees alarming potential for so-called prompt injection attacks driven by the large language models that power AI chatbots. Improve AI security by red teaming large language models: Cyber attacks such as prompt injection pose significant security risks to LLMs, but implementing red teaming strategies can test models’ resistance to various cyber threats. This, according to Sutton, means anybody who wants to disrupt a corporate IT system could look at how they could use an indirect prompt injection attack to insert instructions hidden in normal business correspondence. If an employee is interacting directly with the model and the harmful instructions have found their way into the corporate AI system, then the model may present harmful or misleading content to that person. For example, he said people who submit bids for new project work could provide instructions hidden in their bid, knowing that large language model will be used to summarise the text of their submission, which could be used to influence their bid more positively than rival bids, or instruct the LLM to ignore other bids. For Sutton, this means there is quite a broad range of people that have the means to influence an organisation’s tender process. “You don’t need to be a high-level programmer to put in things like that,” he said. From an IT security perspective, Sutton said an indirect prompt injection attack means people need to be cognisant as to the information being provided to the AI system, since this data is not always reliable. Generally, the output from an LLM is an answer to a query followed by additional contextual information, that shows the users how the information is referenced to output the answer. Sutton pointed out that people should question the reliability of this contextual information, but noted that it would be unrealistic and undermine the usefulness of an LLM if people had to check the context every single time it generated a response. In The Current Issue: Interview: Sarah Hawkins, CEO, National Family Mediation Government faces claims of serious security and data protection problems in One Login digital ID Download Current Issue Brunel versus Putin: next steps for UK tech sector need to be transformational – Data Matters What to expect from Nutanix .NEXT 2025 – CW Developer Network View All Blogs