What are best practices to prevent a tool from being invoked with malicious or malformed input?
Quick Answer
Treat every tools/call argument as untrusted input, exactly like a public HTTP API endpoint would, because it ultimately traces back to model output that can be influenced by adversarial prompts or compromised context. Concretely: validate arguments against the JSON Schema before executing anything; never interpolate arguments directly into a shell command, SQL query, or file path (use parameterized queries, allow-lists, and path-canonicalization checks instead); enforce the server's own authorization on top of whatever the argument says (don't let an account_id argument alone decide whose data to touch — verify the authenticated session actually owns that account); and cap resource-intensive operations (query size limits, rate limits, timeouts) so a single tool call can't be used to exhaust resources or exfiltrate an unbounded amount of data.
Detailed Answer
The arguments in a tools/call request ultimately came from an LLM's output. That output was shaped by conversation content the model read, and some of that content may be attacker-controlled: a malicious webpage, a poisoned document, a crafted email. Treat tool arguments with the same suspicion as a public API request body.
Concrete failure modes to close off
# Vulnerable: argument interpolated straight into a shell command
def run_lookup(hostname: str):
os.system(f"nslookup {hostname}") # hostname="; rm -rf / #" is catastrophic
# Vulnerable: argument interpolated straight into SQL
def get_user(user_id: str):
db.execute(f"SELECT * FROM users WHERE id = {user_id}") # classic SQL injection
# Safer: parameterized query, no shell string interpolation
def run_lookup(hostname: str):
if not HOSTNAME_RE.match(hostname):
raise ToolExecutionError("Invalid hostname format")
return subprocess.run(["nslookup", hostname], capture_output=True)
def get_user(user_id: str):
return db.execute("SELECT * FROM users WHERE id = ?", (user_id,))
A checklist for every tool handler
- Schema-validate first. Reject anything that doesn't match
inputSchemabefore touching business logic. - Never trust an argument as an authorization decision by itself. If a tool takes
account_id, still check server-side that the authenticated caller actually has access to that account. Don't let the model's argument alone decide whose data gets returned or modified. - Avoid raw string interpolation into shells, SQL, or file paths. Use parameterized APIs, allow-lists of permitted values, and path canonicalization for anything path-like — resolve
..traversal, restrict to a known root directory. - Bound the blast radius. Rate-limit calls, cap result sizes (a "list all records" tool shouldn't dump an entire multi-million-row table in one call), and set execution timeouts on anything that shells out or hits a slow external system.
- Log tool invocations (arguments and results, redacting secrets) so a suspicious pattern of calls is auditable after the fact.
- Treat the handler as a public-facing API endpoint at all times, even though the "client" calling it is an LLM rather than a browser.