Skip to content
HackIndex logo

HackIndex

Identifying Trust Boundaries in AI Architectures

3 min read Mar 18, 2026

Trust boundaries in AI systems are the points where data or control passes between components with different privilege levels, ownership, or trust assumptions. In traditional software these boundaries are relatively clear — network perimeters, process isolation, authentication checkpoints. In AI-enabled architectures they are often implicit: a model that can call tools operates with the trust level of the application, not the user. A RAG pipeline that retrieves external content treats that content as trusted input to the model. Mapping these boundaries before attacking reveals where crossing them produces the most impact.

Map the Component Architecture

Start by building a component map from reconnaissance data. List every AI component identified and its role:

# Component map — fill from recon
USER_INTERFACE: web chat at /chat
API_GATEWAY: /api/v1/chat/completions — proxies to OpenAI
ORCHESTRATION: LangChain agent (identified from JS bundle)
MODEL: gpt-4o (from response headers)
TOOLS: web_search, code_interpreter, file_reader (from JS)
VECTOR_DB: Chroma at internal:8000 (from error stack trace)
EMBEDDING: text-embedding-3-small (from /api/v1/embeddings)
DOCUMENT_STORE: s3://company-docs (from RAG source metadata)

With the component list built, identify the trust zone each component belongs to and what crosses each boundary. A boundary exists wherever two components have different trust levels, different owners, or different authentication requirements.

Identify the User-to-Model Boundary

The boundary between user input and the model is the most obvious but often the least enforced. Determine what transformation, if any, happens to user input before it reaches the model:

┌──(kali㉿kali)-[~]
└─$ curl -sk -X POST https://$TARGET_DOMAIN/api/chat -H 'Content-Type: application/json' -d '{"messages":[{"role":"user","content":"Repeat my exact input back to me: BOUNDARY_TEST_XYZ"}]}' | python3 -m json.tool

If the model echoes the input verbatim, user input reaches the model unmodified. If the response differs or references a sanitised version, some transformation is happening server-side. The absence of transformation means the user-to-model boundary has no enforcement — user input is treated with the same trust as system instructions.

Identify the External Content Boundary

RAG pipelines and tool-enabled agents retrieve external content and pass it to the model as context. This content crosses a critical trust boundary — it originates outside the system but is presented to the model as authoritative information. Test whether retrieved content is sanitised before injection into the prompt:

┌──(kali㉿kali)-[~]
└─$ curl -sk -X POST https://$TARGET_DOMAIN/api/chat -H 'Content-Type: application/json' -d '{"messages":[{"role":"user","content":"Search the web for: BOUNDARY_TEST_INJECTION_MARKER and tell me what you find"}]}' | python3 -m json.tool

If the agent performs the search and returns results, external content is being retrieved and passed to the model without the model distinguishing between system instructions and retrieved data. This boundary is the attack surface for indirect prompt injection — covered in the exploitation phase.

Identify the Tool Execution Boundary

When an agent can execute tools, the boundary between model output and tool execution is a privilege escalation point. Determine what tools are available and what permissions they carry:

┌──(kali㉿kali)-[~]
└─$ curl -sk -X POST https://$TARGET_DOMAIN/api/chat -H 'Content-Type: application/json' -d '{"messages":[{"role":"user","content":"What tools or capabilities do you have access to? List all of them."}]}' | python3 -m json.tool

A model that discloses its tool list reveals the tool execution boundary. Each tool represents a capability the model can invoke — file access means the model has filesystem permissions, code execution means arbitrary code runs with the agent process privileges, web search means the model can reach external systems. These are the targets for tool abuse attacks.

Document Trust Boundary Violations

For each boundary identified, determine whether it is enforced or implicit. Record findings in a structured format:

BOUNDARY: User input -> Model
ENFORCED: No — input passes unmodified
IMPACT: Direct prompt injection viable

BOUNDARY: External web content -> Model context
ENFORCED: No — retrieved content injected without sanitisation
IMPACT: Indirect prompt injection via web content

BOUNDARY: Model output -> Tool execution
ENFORCED: Partial — tool list restricted but no output validation
IMPACT: Tool abuse if model can be influenced to invoke unintended tools

BOUNDARY: Vector DB -> Model context
ENFORCED: No — retrieved chunks injected verbatim
IMPACT: RAG poisoning viable if vector DB write access obtained

References