Talking about AI: Definitions Artificial Intelligence (AI) — AI refers to the simulation of human intelligence in machines, enabling them to perform tasks that typically require human intelligence, such as decision-making and problem-solving. AI is the broadest concept in this field, encompassing various technologies and methodologies, including Machine Learning (ML) and Deep Learning. Machine Learning (ML) — ML is a subset of AI that focuses on developing algorithms and statistical models that allow machines to learn from and make predictions or decisions based on data. ML is a specific approach within AI, emphasizing data-driven learning and improvement over time. Deep Learning (DL) — Deep Learning is a specialized subset of ML that uses neural networks with multiple layers to analyze and interpret complex data patterns. This advanced form of ML is particularly effective for tasks such as image and speech recognition, making it a crucial component of many AI applications. Large Language Models (LLM) — LLMs are a type of AI model designed to understand and generate human-like text by being trained on extensive text datasets. These models are a specific application of Deep Learning, focusing on natural language processing tasks, and are integral to many modern AI-driven language applications. Generative AI (GenAI) — GenAI refers to AI systems capable of creating new content, such as text, images, or music, based on the data they have been trained on. This technology often leverages LLMs and other Deep Learning techniques to produce original and creative outputs, showcasing the advanced capabilities of AI in content generation. Overview: AI for Good and Bad Almost daily now we watch the hallowed milestone of the "Turing Test" slip farther and farther into an almost naïve irrelevance, as computer interfaces have evolved from being comparable to human language, to similar, to indistinguishable, to arguably superior [1]. The development of large language models (LLMs) began with natural language processing (NLP) advancements in the early 2000s, but the major breakthrough came with Ashish Vaswani's 2017 paper, "Attention is All You Need." This allowed for training larger models on vast datasets, greatly improving language understanding and generation. Like any technology, LLMs are neutral and can be used by both attackers and defenders. The key question is, which side will benefit more, or more quickly? Let's dive into that question in a bit more detail. This is but an excerpt of our coverage in the Security Navigator 2025, but it covers some of the main points that should be relevant to everyone who works in a security- or technology context. If you want to read more on 'Prompt Injection' techniques or how AI is productively used in security technology I invite you to get the full report! AI in defense operations May improve general office productivity and communication May improve search, research and Open-Source Intelligence May enable efficient international and cross-cultural communications May assist with collation and summarization of diverse, unstructured text datasets May assist with documentation of security intelligence and event information May assist with analyzing potentially malicious emails and files May assist with identification of fraudulent, fake or deceptive text, image or video content. May assist with security testing functions like reconnaissance and vulnerability discovery. AI in one form or another has long been used in a variety of security technologies. By way of example: Intrusion Detection Systems (IDS) and Threat Detection. Security vendor Darktrace, employs ML to autonomously detect and respond to threats in real-time by leveraging behavioral analysis and ML algorithms trained on historical data to flag suspicious deviations from normal activity. Phishing Detection and Prevention. ML models are used in products like Proofpoint and Microsoft Defender that identify and block phishing attacks utilizing ML algorithms to analyze email content, metadata, and user behavior to identify phishing attempts.Endpoint Detection and Response (EDR). EDR offerings like CrowdStrike Falcon leverage ML to identify unusual behavior and detect and mitigate cyber threats on endpoints.Microsoft Copilot for Security. Microsoft's AI-powered solution is designed to assist security professionals by streamlining threat detection, incident response, and risk management by leveraging generative AI, including OpenAI's GPT models.AI in offensive operations May improve general office productivity and communication for bad actors as well May improve search, research and Open-Source Intelligence May enable efficient international and cross-cultural communications May assist with collation and summarization of diverse, unstructured text datasets (like social media profiles for phishing/spear-phishing attacks) May assist with attack processes like reconnaissance and vulnerability discovery. May assist with the creation of believable text for cyber-attack methods like phishing, waterholing and malvertising. Can assist with the creation of fraudulent, fake or deceptive text, image or video content. May facilitate accidental data leakage or unauthorized data access May present a new, vulnerable and attractive attack surface. Real-world examples of AI in offensive operations have been relatively rare. Notable instances include MIT's Automated Exploit Generation (AEG)[2] and IBM's DeepLocker[3], which demonstrated AI-powered malware. These remain proof-of-concepts for now. In 2019, our research team presented two AI-based attacks using Topic Modelling[4], showing AI's offensive potential for network mapping and email classification. While we haven't seen widespread use of such capabilities, in October 2024, our CERT reported[5] that the Rhadamanthys Malware-as-a-Service (MaaS) incorporated AI to perform Optical Character Recognition (OCR) on images containing sensitive information, like passwords, marking the closest real-world instance of AI-driven offensive capabilities. Security Navigator 2025 is Here - Download Now The newly released Security Navigator 2025 offers critical insights into current digital threats, documenting 135,225 incidents and 20,706 confirmed breaches. More than just a report, it serves as a guide to navigating a safer digital landscape. What's Inside?# 📈 In-Depth Analysis: Statistics from CyberSOC, Vulnerabilitiy scanning, Pentesting, CERT, Cy-X and Ransomware observations from Dark Net surveillance. 🔮 Future-Ready: Equip yourself with security predictions and stories from the field. 👁️ Security deep-dives: Get briefed on emerging trends related to hacktivist activities and LLMs/Generative AI. Stay one step ahead in cybersecurity. Your essential guide awaits! 🔗 Get Your Copy Now LLMs are increasingly being used offensively, especially in scams. A prominent example is the UK engineering group Arup[6], which reportedly lost $25 million to fraudsters who used a digitally cloned voice of a senior manager to order financial transfers during a video conference. Does AI drive threats? For systematically considering the potential risk from LLM technologies, we examine four perspectives: the risk of not adopting LLMs, existing AI threats, new threats specific to LLMs, and broader risks as LLMs are integrated into business and society. These aspects are visualized in the graphic below: Branch 1: The Risk of Non-adoption Many clients we talk to feel pressure to adopt LLMs, with CISOs particularly concerned about the "risk of non-adoption," driven by three main factors: Efficiency loss: Leaders believe LLMs like Copilot or ChatGPT will boost worker efficiency and fear falling behind competitors who adopt them. Opportunity loss: LLMs are seen as uncovering new business opportunities, products, or market channels, and failing to leverage them risks losing a competitive edge. Marketability loss: With AI dominating discussions, businesses worry that not showcasing AI in their offerings will leave them irrelevant in the market. These concerns are valid, but the assumptions are often untested. For example, a July 2024 survey by the Upwork Research Agency [7] revealed that "96% of C-suite leaders expect AI tools to boost productivity." However, the report points out, "Nearly half (47%) of employees using AI say they have no idea how to achieve the productivity gains their employers expect, and 77% say these tools have actually decreased their productivity and added to their workload. The marketing value of being "powered by AI" is also still debated. A recent FTC report notes that consumers have voiced concerns about AI's entire lifecycle, particularly regarding limited appeal pathways for AI-based product decisions. Businesses must consider the true costs of adopting LLMs, including direct expenses like licensing, implementation, testing, and training. There's also an opportunity cost, as resources allocated to LLM adoption could have been invested elsewhere. Security and privacy risks need to be considered too, alongside broader economic externalities—such as the massive resource consumption of LLM training, which requires significant power and water usage. According to one article [8], Microsoft's AI data centers may consume more power than all of India within the next six years. Apparently "They will be cooled by millions upon millions of gallons of water". Beyond resource strain, there are ethical concerns as creative works are often used to train models without creators' consent, affecting artists, writers, and academics. Additionally, AI concentration among a few owners could impact business, society, and geopolitics, as these systems amass wealth, data, and control. While LLMs promise increased productivity, businesses risk sacrificing direction, vision, and autonomy for convenience. In weighing the risk of non-adoption, the potential benefits must be carefully balanced against the direct, indirect, and external costs, including security. Without a clear understanding of the value LLMs may bring, businesses might find the risks and costs outweigh the rewards. Branch 2: Existing Threats From AI In mid October 2024, our "World Watch" security intelligence capability published an advisory that summarized the use of AI by offensive actors as follows: "The adoption of AI by APTs remains likely in early stages but it is only a matter of time before it becomes more widespread." One of the most common ways state-aligned and state-sponsored threat groups have been adopting AI in their kill chains is by using Generative AI chatbots such as ChatGPT for malicious purposes. We assess that these usages differ depending on each group's own capabilities and interests. North Korean threat actors have been allegedly leveraging LLMs to better understand publicly reported vulnerabilities [9], for basic scripting tasks and for target reconnaissance (including dedicated content creation used in social engineering). Iranian groups were seen generating phishing emails and used LLMs for web scraping [10]. Chinese groups such as Charcoal Typhoon abused LLMs for advanced commands representative of post-compromise behavior [10]. In October 9, OpenAI disclosed [11] that since the beginning of the year it had disrupted over 20 ChatGPT abuses aimed at debugging and developing malware, spreading misinformation, evading detection, and launching spear-phishing attacks. These malicious usages were attributed to Chinese (SweetSpecter) and Iranian threat actors (CyberAv3ngers and Storm-0817). The Chinese cluster SweetSpecter (tracked as TGR-STA-0043 by Palo Alto Networks) even targeted OpenAI employees with spear-phishing attacks. Recently, state-sponsored threat groups have also been observed carrying out disinformation and influence campaigns targeting the US presidential election for instance. Several campaigns attributed to Iranian, Russian and Chinese threat actors leveraged AI tools to erode public trust in the US democratic system or discredit a candidate. In its Digital Defense Report 2024, Microsoft confirmed this trend, adding that these threat actors were leveraging AI to create fake text, images and videos. Cybercrime In addition to leveraging legitimate chatbots, cybercriminals have also created "dark LLMs" (models trained specifically for fraudulent purposes) such as FraudGPT, WormGPT and DarkGemini. These tools are used to automate and enhance phishing campaigns, help low-skilled developers create malware, and generate scam-related content. They are typically advertised on the DarkWeb and Telegram, with an emphasis on the model's criminal function. Some financially-motivated threat groups are also adding AI to their malware strains. A recent World Watch advisory on the new version of the Rhadamanthys infostealer describes new features relying on AI to analyze images that may contain important information, such as passwords or recovery phrases. In our continuous monitoring of cybercriminal forums and marketplaces we observed a clear increase in malicious services supporting social-engineering activities, including: Deepfakes, notably for sextortion and romance schemes. This technology is becoming more convincing and less expensive over time. AI-powered phishing and BEC tools designed to facilitate the creation of phishing pages, social media contents and email copies. AI-powered voice phishing. In a report published on July 23, Google revealed [12] how AI-powered vishing (or voice-spoofing), facilitated by commodified voice synthesizers, was an emerging threat. Vulnerability exploitation AI still faces limits when used to write exploit code based on a CVE description. If the technology improves and becomes more readily available, it will likely be of interest to both cybercriminals and state-backed actors. An LLM capable of autonomously finding a critical vulnerability, writing and testing exploit code and then using it against targets, could deeply impact the threat landscape. Exploit development skills could thus become accessible to anyone with access to an advanced AI model. The source code of most products is fortunately not readily available for training such models, but open source software may present a useful testcase. Branch 3: New Threats from LLMs The new threats emerging from widespread LLM adoption will depend on how and where the technology is used. In this report, we focus strictly on LLMs and must consider whether they are in the hands of attackers, businesses, or society at large. For businesses, are they consumers of LLM services or providers? If a provider, are they building their own models, sourcing models, or procuring full capabilities from others? Each scenario introduces different threats, requiring tailored controls to mitigate the risks specific to that use case. Threats to Consumers A Consumer uses GenAI products and services from external providers, while a Provider creates or enhances consumer-facing services that leverage LLMs, whether by developing in-house models or using third-party solutions. Many businesses will likely adopt both roles over time. It's important to recognize that employees are almost certainly already using public or local GenAI for work and personal purposes, posing additional challenges for enterprises. For those consuming external LLM services, whether businesses or individual employees, the primary risks revolve around data security, with additional compliance and legal concerns to consider. The main data-related risks include: Data leaks: Workers may unintentionally disclose confidential data to LLM systems like ChatGPT, either directly or through the nature of their queries. Hallucination: GenAI can produce inaccurate, misleading, or inappropriate content that employees might incorporate into their work, potentially creating legal liability. When generating code, there's a risk it could be buggy or insecure [13]. Intellectual Property Rights: As businesses use data to train LLMs and incorporate outputs into their intellectual property, unresolved questions about ownership could expose them to liability for rights violations. The outputs of GenAI only enhance productivity if they are accurate, appropriate, and lawful. Unregulated AI-generated outputs could introduce misinformation, liability, or legal risks to the business. Threats to providers An entirely different set of threats emerge when businesses choose to integrate LLM into their own systems or processes. These can be broadly categorized as follows: Model Related Threats A trained or tuned LLM has immense value to its developer and is thus subject to threats to its Confidentiality, Integrity and Availability. In the latter case, the threats to proprietary models include: Theft of the model. Adversarial "poisoning" to negatively impact the accuracy of the model. Destruction or disruption of the model. Legal liability that may emerge from the model producing incorrect, misrepresentative, misleading, inappropriate or unlawful content. We assess, however, that the most meaningful new threats will emerge from the increased attack surface when organizations implement GenAI within their technical environments. GenAI as Attack Surface GenAI are complex new technologies consisting of millions of lines of code that expand the attack surface and introduce new vulnerabilities. As general GenAI tools like ChatGPT and Microsoft Copilot become widely available, they will no longer offer a significant competitive advantage by themselves. The true power of LLM technology lies in integrating it with a business's proprietary data or systems to improve customer services and internal processes. One key method is through interactive chat interfaces powered by GenAI, where users interact with a chatbot that generates coherent, context-aware responses. To enhance this, the chat interface must leverage capabilities like Retrieval-Augmented Generation (RAG) and APIs. GenAI processes user queries, RAG retrieves relevant information from proprietary knowledge bases, and APIs connect the GenAI to backend systems. This combination allows the chatbot to provide contextually accurate outputs while interacting with complex backend systems. However, exposing GenAI as the security boundary between users and a corporation's backend systems, often directly to the Internet, introduces a significant new attack surface. Like the graphical Web Application interfaces that emerged in the 2000's to offer easy, intuitive access to business clients, such Chat Interfaces are likely to transform digital channels. Unlike graphical web interfaces, GenAI's non-deterministic nature means that even its developers may not fully understand its internal logic, creating enormous opportunity for vulnerabilities and exploitation. Attackers are already developing tools to exploit this opacity, leading to potential security challenges similar to those seen with early web applications, that are still plaguing security defenders today. Tricking LLMs out of their 'guardrails' The Open Web Application Security Project (OWASP) has identified "Prompt Injection" as the most critical vulnerability in GenAI applications. This attack manipulates language models by embedding specific instructions within user inputs to trigger unintended or harmful responses, potentially revealing confidential information or bypassing safeguards. Attackers craft inputs that override the model's standard behavior. Tools and resources for discovering and exploiting prompt injection are quickly emerging, similar to the early days of web application hacking. We expect that Chat Interface hacking will remain a significant cybersecurity issue for years, given the complexity of LLMs and the digital infrastructure needed to connect chat interfaces with proprietary systems. As these architectures grow, traditional security practices—such as secure development, architecture, data security, and Identity & Access Management—will become even more crucial to ensure proper authorization, access control, and privilege management in this evolving landscape. When the "NSFW" AI chatbot site Muah.ai was breached in October 2024, the hacker described the platform as "a handful of open-source projects duct-taped together." Apparently, according to reports, "it was no trouble at all to find a vulnerability that provided access to the platform's database". We predict that such reports will become commonplace in the next few years. Conclusion: more of the same is not a new dimension Like any powerful technology, we naturally fear the impact LLMs could have in the hands of our adversaries. Much attention is paid to the question of how AI might "accelerate the threat. The uncertainty and anxiety that emerges from this apparent change in the threat landscape is of course exploited to argue for greater investment in security, sometimes honestly, but sometimes also duplicitously. However, while some things are certainly changing, many of the threats being highlighted by alarmists today pre-exist LLM technology and require nothing more of us than to keep consistently doing what we already know to do. For example, all the following threat actions, whilst perhaps enhanced by LLMs, have already been performed with the support of ML and other forms of AI [14] (or indeed, without AI at all): Online Impersonation Cheap, believable phishing mails and sites Voice fakes Translation Predictive password cracking Vulnerability discovery Technical hacking The notion that adversaries may execute such activities more often or more easily is a cause for concern, but it does not necessarily require a fundamental shift in our security practices and technologies. LLMs as an attack surface on the other hand are vastly underestimated. It is crucial that we learn the lessons of previous technology revolutions (like web applications and APIs) so as not to repeat them by recklessly adopting an untested and somewhat untestable technology at the boundary between open cyberspace and our critical internal assets. Enterprises are well advised to be extremely cautious and diligent in weighing up the potential benefits of deploying a GenAI as an interface, with the potential risks that such a complex, untested technology will surely introduce. Essentially we face at least the same access- and data safety issues we already know from the dawn of the cloud age and subsequent erosion of the classic company perimeter. Despite the ground-breaking innovations we're observing, security "Risk" is still comprised fundamentally from the product of Threat, Vulnerability and Impact, and an LLM cannot magically create these if they aren't already there. If those elements are already there, the risk a business has to deal with is largely independent of the existence of AI. This is just an excerpt of the research we did on AI and LLMs. To read the full story and more detailed advisory, as well as expert stories about how prompt injections work to manipulate LLMs and work outside their safety guardrails, or how defenders use AI to detect subtle signals of compromise in vast networks: it's all in the Security Navigator 2025. Head over to the download page and get your copy! Found this article interesting? This article is a contributed piece from one of our valued partners. Follow us on Twitter  and LinkedIn to read more exclusive content we post.