Security & Sandboxing

Three Risk Categories: Misuse, Misbehavior, External Attack

Security isn't just adding a prompt layer—it requires structural design at the architecture level. Here we introduce a systematic security classification framework.

Security Classification Framework

Every AI product faces a different threat model, but all risks fall into three categories. Understanding these three risk types is the first step to designing a secure architecture.

MISUSE
User Misuse
Users intentionally make the Agent do things it shouldn't. This is an active threat from the user side.
  • Using the Agent to generate phishing emails
  • Inducing the Agent to execute malicious code
  • Using the Agent to access unauthorized data
  • Bypassing safety limits through jailbreak attacks
MISBEHAVIOR
Model Misbehavior
The model acts wrongly on its own: no one told it to, but it did something it shouldn't.
  • Overreach: user only asked to view a file, but the model modified it on its own
  • Hallucination-driven action: executed a real operation based on fabricated information
  • Permission overstep: model tries to access resources outside the current task
  • Can't stop: Agent enters an infinite loop
EXTERNAL ATTACK
External Attack
Third parties manipulate the Agent's behavior by injecting malicious content. The attack originates from data sources—the user may be completely unaware.
  • Prompt Injection: attack instructions embedded in web pages/documents
  • Supply chain attack: malicious MCP server returns tampered data
  • Data poisoning: backdoors planted in training data
  • Indirect injection: instructions injected through emails/files that the Agent reads
Dual-Layer Containment Strategy

MODEL LAYER Model-Layer Defense

Training the model to naturally prefer safe behaviors—like cultivating an employee with good values.
  • RLHF/Constitutional AI trains safe preferences
  • Model learns to refuse dangerous requests
  • Model proactively asks the user when uncertain
  • Follows the principle of least privilege

ENVIRONMENT LAYER Environment-Layer Defense

System architecture that makes dangerous operations impossible—like locking a warehouse, not relying on the employee's conscience.
  • Sandbox isolation: code executes in a restricted environment
  • Permission controls: authorization scoped to task granularity
  • Approval mechanism: high-risk operations require human confirmation
  • Network isolation: restricts the Agent's network access scope
The two are complementary—neither can be omitted. The model layer makes the Agent want to do the right thing; the environment layer makes it impossible to do the wrong thing even if it tries. Telling the model "don't do bad things" via Prompt alone is insufficient—you also need the architecture to make bad things structurally impossible.
MCP's Dual Risks

New Attack Surface Introduced by Model Context Protocol

SUPPLY CHAIN RISK
Supply Chain Risk
Untrusted MCP servers can inject malicious content. The Agent trusts tool descriptions and data returned by MCP, but these may already be tampered with. A malicious MCP server can manipulate the Agent's behavior by altering tool descriptions: the Agent thinks it's using a "search files" tool, but is actually executing a delete operation.
PROMPT INJECTION
Injection Attack
Content returned by MCP may contain attack instructions. Even if the MCP server itself is not malicious, data it returns (e.g., content scraped from web pages) may contain Prompt Injection attacks. When the Agent processes this data, it may be convinced to perform unintended operations.
MCP fundamentally expands the Agent's attack surface. Each additional MCP server is another potential data injection entry point. Product designers need to vet every MCP integration the same way they vet third-party SDKs—trust but verify.
Auto Mode in Practice

Classifier + Sandbox: High Autonomy, Low Risk

~83%
Reduction in permission prompts
2
Core components
Auto Mode achieves the balance of high autonomy + low risk through two core components:
Classifier
Judges whether each operation is safe
+
Sandbox
No damage even if the classifier misjudges
=
High Autonomy + Low Risk
The classifier quickly assesses the risk level of each operation: safe operations execute directly; suspicious ones trigger a confirmation prompt. The sandbox acts as a second line of defense, ensuring that even if the classifier misjudges, code execution causes no real harm to the system. Together, they reduced user confirmation prompts by ~83% while maintaining security.
Security isn't just adding a prompt layer—it requires structural design. Understanding the three risk categories (misuse, misbehavior, attack) and building defenses at both the model layer and the environment layer is what lets an Agent run safely in real production environments.