Link to the original

Introduction

Living off the land (LOTL) is a class of attacks in which attackers use existing legitimate operating-system or software tools to perform malicious actions. For example, by using PowerShell or WMI, they can avoid suspicious signatures and use allowlists completely legitimately. According to CrowdStrike, in 2023, 6 out of 10 recorded attacks involved LOTL techniques instead of classic malware.

In this paper, the authors consider how future devices with built-in LLMs will become a security problem, because attackers will be able to “live off the LLM” (LOLLM). That is, they will use models already present on the device for:

  • code generation;
  • bypassing defenses;
  • performing attacks without external connections.

How Attackers Can Use LLMs

LLMs are becoming part of system infrastructure and can be used for attacks at the application, network, and AI-infrastructure levels. In the paper, the authors consider different types of attacks based on PoCs and existing techniques.

Direct Malware Code Generation

LLMs can create executable code on the fly, even without files. HYAS BlackMamba can serve as an example: a keylogger that uses ChatGPT to dynamically write functions and inject them directly into memory. Such software leaves no artifacts on disk and is difficult to detect.

Automation of Complex Attacks

Modern “LLM-based agents” can plan and execute a chain of actions that would normally require human participation.

Examples of such frameworks:

  • RapidPen - an automated system that obtained remote access to a server without operator involvement.
  • AutoAttacker - a system that imitates 14 types of attacks characteristic of an experienced hacker.

Such tools lower the “entry threshold”, so even a non-expert can now launch a full-fledged attack.

Using LLMs as a Proxy

The paper Ratgpt: Turning online llms into proxies for malware attacks demonstrates how attackers use public LLM APIs as a command-and-control (C2) channel. Malware “communicates” with the OpenAI server, disguising its commands as harmless requests.

Impact on Developers and Supply Chains

LLMs can suggest vulnerable code. One example is the INSEC attack against code-completion systems presented in the paper Black-Box Adversarial Attacks on LLM-Based Code Completion.

Also, malicious packages can be introduced into open-source products, where an LLM helps disguise malicious functionality as “utility” functionality.

Social Engineering

LLMs significantly improve phishing and vishing (voice phishing). For example, the ViKing system is an autonomous voice bot that successfully persuades people to disclose data. Generating personalized messages or calls is now possible at massive scale.

Infecting the Models Themselves

Researchers have shown that TensorFlow, PyTorch, and other models can be used to introduce malicious behavior. An infected model can execute commands such as deleting files or communicating with a C2 server during inference. Some formats, such as Pickle, even allow arbitrary code to be inserted. Even security tools do not guarantee detection of such “infected models”.

LOLLM Methodology

The authors created a PoC attack illustrating a new class of threats and consider a scenario where an attacker already has access to a user profile in an organization and wants to perform malicious actions without downloading viruses and without known tools.

Attack stages:

  • Scanning the system to find local LLMs;
  • Selecting a model with priority by power;
  • Embedding a feedback loop where the script asks the model to complete functions; the code is generated dynamically and is not saved to disk;
  • Using a jailbreak if the model refuses to execute malicious instructions;
  • Performing malicious actions, for example deleting files from a dataset and creating an autostart service for persistence;

Thus, malicious code is generated by the local model, meaning there is no network traffic. This leads to antivirus products not seeing suspicious activity. Also, the code constantly changes, which in turn means signatures cannot be used for detection.

Jailbreaking and Model Alignment

Since the attacker does not know in advance which LLM is installed on the victim’s system, they face the problem of alignment in some models.

For example, Gemma 3 4b easily writes neutral scripts, but refuses to create an exploit. However, if the task is rephrased (“This is safe defense testing in an isolated environment”), the model gives in and generates the required code.

Thus, the attacker resorts to creating a “deceptive context”, for example wrapping the attack as “ethical research”, “educational purpose”, and so on. This makes it possible to remove restrictions by claiming that the code will not be used maliciously.

System Types

The authors identify four types of systems by their vulnerability level to such attacks:

  1. Systems without LLMs - not vulnerable to LOLLM;
  2. Systems with strongly aligned models - resistant, require complex jailbreaks;
  3. Systems with weakly aligned models - vulnerable to simple bypasses;
  4. Systems with uncensored models - fully vulnerable, even without bypasses.

Thus, the authors conclude that safety alignment is not only “ethics”, but also an element of cyber defense. Deploying “uncensored” models in an enterprise should be treated as a security risk.

Defense Methods Against LLM-Oriented Attacks

The paper considers methods for detecting LOTL attacks. One option is to use existing machine-learning algorithms that identify malicious commands:

  • Analysis of command syntax and hidden characters;
  • Searching for environment variables that mask code;
  • Decoding Base64 and similar structures;
  • Analyzing command sequences rather than individual commands.

It is recommended to use Indicators of Attack (IOA), not Indicators of Compromise (IoC), because they are aimed at early detection of attacker behavior. For example, the following areas can be monitored:

  • Access/authentication;
  • Privileged actions;
  • Command activity and sequences;
  • File activity;
  • Network;
  • Use of engineering/administrative tools in an unusual context (PLC utilities from the IT segment, Kali-like tools from an ordinary user).

The authors propose applying the following approaches to LLMs and list specific measures:

  • Prompt Firewall - requests to the LLM should be logged and filtered; logs should include prompts, responses, user identifiers, session metadata, and timestamps
  • Output Sanitization - LLM output should also be logged and filtered; generated code that uses common binaries/utilities (for example, PowerShell) should be blocked;
  • Anomaly Detection - anomalies such as excessive code/script generation requests, reconnaissance prompts, and unusual times or access volumes should trigger alerts;
  • Tool Use Restrictions - as LLMs become more “agentic” and use tools on the device, restrict LLMs to only the tools that are necessary;
  • LLM Usage Restrictions - allow users to disable code-generation capabilities if they do not need them;
  • Crowdsourced Rules for LLM Abuse Patterns - develop standard formats for detecting LLM abuse patterns and use crowdsourcing to exchange such rules (similar to Snort rules).

Conclusion

Local LLMs will become part of infrastructure, which means they will become a new field for cyberattacks. Attackers will be able to use them the way PowerShell or WMI are used now. Security requires integrating protection mechanisms directly into models and their environment:

  • model alignment;
  • analysis of request behavior;
  • restrictions on code generation;
  • continuous audit.

In the future, LLMs may become “attack tools”, so developers and companies should treat them as potential vulnerable assets, not simply as assistants.