Hallucinated tool arguments: the right tool with the wrong values

Function-calling models are trained to produce arguments that fit a tool's schema. When the information they need is missing, they do not always stop and ask; sometimes they fill the gap with something plausible. Public evaluations of function calling, such as the Berkeley Function Calling Leaderboard and the Gorilla work behind it, exist precisely because models make these errors: wrong functions, wrong or invented argument values, and calls that should not have been made at all.

In a live agent that looks like:

• Invented identifiers. The user asked to "cancel my last order"; the agent calls orders.cancel with order_id: "ORD-10492", an id that appears nowhere in the conversation.
• Unsupported amounts. The ticket says the customer was charged twice; the agent refunds a round number that does not match either charge.
• Guessed recipients. "Send the report to Sam" becomes an email to an address the model constructed from a first name and a domain.
• Plausible paths. A cleanup task deletes /var/data/old, a directory the user never named.

Each of these passes a JSON schema. The types are right; the values are fiction.

Most deterministic controls look at the tool and the shape of its arguments: is orders.cancel allowed for this agent, is amount a positive number below a limit, is the path inside the working directory. Those rules are necessary, and they catch a lot. They cannot tell you whether the value came from the user or from the model's imagination, because that depends on the conversation, not on the call.

This is also why hallucinated arguments matter for security, not just quality. An argument that is not grounded in the user's request may have come from somewhere else, such as an instruction hidden in a web page or a tool result. The args_grounded head and the injected head look at related questions from different angles; see prompt injection through tool results.

Every check answers nine questions in one pass. One of them, args_grounded, estimates whether the action's arguments are supported by what the user asked: no invented ids, amounts or recipients. By default, a score below 0.5 turns the verdict into ask, even when the action is otherwise low-risk, because a well-intended action on the wrong target is still the wrong action.

The guard can only judge grounding against what it sees. A check that contains the action alone gives it nothing to compare with. Send the user's side too:

{
  "action": { "tool": "billing.refund", "args": { "charge_id": "ch_3Qx9", "amount": 5000, "currency": "usd" } },
  "intent": "Refund the duplicate charge the customer reported",
  "user_message": "I was charged $19 twice on March 3rd, please refund one of them.",
  "trigger": "user_request",
  "constraints": ["Refund at most the amount of one reported charge"]
}

Here the amount (5000 cents, $50) is not supported by the message ($19), so you would want this to come back as ask. With only the action, the same call looks like an ordinary refund.

Useful fields, in rough order of value: intent (what the user wants done), user_message (their latest words, when different), and conversation (recent turns, oldest first, most recent ones only). If the value came from a previous tool result, for example an order lookup, include that turn so the guard can see where the id came from.

Grounding checks are probabilistic and work best alongside cheap deterministic ones:

• Validate against your own data. If an order id does not exist or belongs to another customer, reject it before any model sees it. A lookup is exact; a guard is an estimate.
• Bound the values. Refunds no larger than the original charge, recipients inside allowed domains, paths inside the workspace.
• Prefer references over free text. Tools that take an id selected from a previous lookup leave less room for invention than tools that take a free-form string.
• Ask the user when information is missing. Prompting the agent to ask rather than guess reduces the problem at the source.

The guard covers the gap between these: values that are valid, in range and exist, but still are not what the user asked for. It reduces the rate at which such calls get through; it does not eliminate it.