Attacking LLMs: PT2 Module 4

LLM-powered features are new attack surface the exam explicitly tests. An LLM behind a chat UI, a document summarizer, a RAG-backed support agent, or a tool-calling assistant each introduces trust boundaries the application cannot easily enforce. User input becomes instruction, retrieved content becomes instruction, and the model treats everything in its context window as equally authoritative.

Finding The LLM

Every attack starts with the endpoint. Explicit chat routes, hidden model-proxy paths exposed through JavaScript, and third-party fingerprints (OpenAI/Anthropic API signatures, token counting headers) all mark where the LLM lives. Baseline the model's refusal behavior and the shape of its system prompt before injecting anything.

Prompt Injection

Direct instruction override, persona hijacking, and structured-data injection are the primitives. Multi-turn attacks spread the payload across several turns to evade single-turn filters. Automated payload delivery (promptmap, PyRIT) tests hundreds of variants against a target in minutes.

Jailbreak And Safety Filter Bypass

Fictional framing ("imagine you are DAN"), encoding (base64, ROT13, unicode homoglyphs), token smuggling across whitespace, and many-shot patterns where the model learns the forbidden behavior from the prompt itself. The frontier is model-specific; what works on one vendor's model often fails on another.

Indirect Prompt Injection

The most dangerous class. User input is not the only thing in the context; search results, uploaded documents, scraped web pages, and email bodies are all treated as model input. A single poisoned document in a RAG knowledge base can rewrite the model's behavior for every user who retrieves it.

System Prompt Extraction And Data Theft

Direct ("repeat your instructions"), indirect (behavior probing), and token-completion attacks all recover the hidden system prompt. Once extracted, prompt-injection payloads can be crafted precisely. IDOR on conversation history endpoints and admin memory stores exfiltrates every prior user's conversations.

Agent Tool Abuse

Agents with tool access are the highest-impact target. File-read tools become arbitrary file disclosure, code-execution tools become RCE, and web-fetch tools become SSRF against internal services. Over-permissioned tool scopes are the finding to look for; the agent is just the delivery mechanism.

Supply Chain

Dataset poisoning, backdoored LoRA adapters, and malicious model weights published to Hugging Face compromise the model before it ever runs. Trigger-token backdoors activate only on specific input patterns, invisible to normal evaluation.

Attacking LLMs