For a tool-call to succeed, the call must adhere to the tool’s schema.

The simple approach to tool calling is to let the model generate tool calls freely and trust that the model’s training is sufficient to generate correct calls. This is similar to how we trust the model to generate syntactically correct prose.

The simple approach works most of the time because modern models are good at following structured patterns. But success “most of the time” implies that many tool calls can still fail, resulting in context window pollution due to error messages, lower model performance, and increased cost due to the agent having to retry the call one or more times.

To avoid such experiences when using Kimi, Moonshot AI published their K2VV ToolCall benchmark. This benchmark lets providers verify that they serve Kimi with the same quality as the original, including its default behavior of only generating valid tool calls.

K2VV ToolCall’s workload consists of 2000 requests selected to evaluate when the LLM generates tool calls and whether the generated tool calls are syntactically correct (i.e., adhere to the provided schema). The benchmark allows analyzing the model’s quality by quantifying two metrics: trigger similarity and schema accuracy. Trigger similarity considers when the service decides to trigger a tool call, and how similar its decisions are to Moonshot AI’s own service. Schema accuracy quantifies the fraction of tool-calls that adhere to the tool’s schema and can therefore be invoked successfully. For example, when tool-call accuracy is 50%, half of the tool calls generated by the model are malformed.

Schema accuracy usually falls short due to the probabilistic nature of language models. A model generates tokens based on likelihood. It is not running a compiler or type checker. Even with a clear prompt, it can still produce malformed tool calls. This can be caused by, for example, a mismatch between the model’s training data and the current inference harness. Models are trained on many tool-calling datasets, each with its own tools, schemas, and conventions. As a result, a model may generate a tool or field name that it has seen during the training, even if it does not exist in the current production environment. For example, it may generate search_web when the available tool is web_search, or use city_name when the active schema expects location.

An LLM computes for each token the probability that it is the next in the sequence. It then selects the next token by performing a weighted sample. Constrained decoding avoids malformed tool calls by preventing models from sampling tokens that would result in a syntactically invalid tool call. When using constrained decoding, instead of letting the model sampling tokens freely and validating the result afterward, the inference engine guides generation step by step. At each step, only valid tokens are allowed.

Which tokens are allowed is determined by the provided tool schema. vLLM implements this using structural tags from the XGrammar library. The snippet below shows an excerpt of how XGrammar implements constrained decoding in Kimi K2.6. It defines the valid structure of a tool call, including where the call begins, where the index appears, where the arguments begin, and where schema-constrained JSON must be generated.

# Kimi K2.6 tool call syntax:
# <|tool_call_begin|>{function.name}:{index}
# <|tool_call_argument_begin|>{arguments}
# <|tool_call_end|>
TagFormat(
    begin=f"{TOOL_CALL_BEGIN_PREFIX}{name}{TOOL_CALL_SUFFIX}",
    content=SequenceFormat(
        elements=[
            RegexFormat(pattern=r"\d+"),
            ConstStringFormat(value=TOOL_CALL_ARGUMENT_BEGIN),
            JSONSchemaFormat(
                json_schema=parameters,
                any_order=any_order,
                max_whitespace_cnt=max_whitespace_cnt,
            ),
        ]
    ),
    end=TOOL_CALL_END,
)

You can find the full implementation here.

With this definition, generation must follow a valid path:

Invalid paths are blocked during generation. For example, when generation reaches the tool-call index phase, the engine restricts sampling to integer tokens and masks out everything else. The figure below, reproduced from XGrammar’s article, provides a good visualization of how invalid tokens are masked.

Source: Dong, Yixin, et al. "XGrammar: Flexible and efficient structured generation engine for large language models." Proceedings of Machine Learning and Systems 7 (2025). [link]

This changes schema accuracy from a best-effort approach to a runtime guarantee. The model still decides which token to generate, but now the token is selected from a restricted and syntactically-correct set.

Notably, this runtime enforcement adds negligible performance overhead. By precomputing token masks for each phase, and applying them efficiently on the GPU during generation, XGrammar guarantees correct tool calls without trading off significant performance.

Moonshot AI recommends constrained decoding to be turned on and its own API behaves as such. So we initially added constrained decoding in our fork of vLLM. After vLLM added this feature in v0.25, we switched to using this feature from upstream for Kimi and DeepSeek. Unfortunately not long after, we found out that our tool-call accuracy score dropped back to 91%!

After analyzing the problem, we confirmed that vLLM’s implementation of constrained decoding was still correct. Instead, we found that vLLM changed its default behavior for requests that do not explicitly turn constrained decoding on. This is primarily because vLLM is designed to support a wide range of models. Supporting constrained decoding is a per-model task and problems fall into three camps:

• General model issues: For some models, their unconstrained output diverges significantly from the tool's schema or model's tool-calling format. This forces the model to select tokens it otherwise wouldn't have, leading it off course [1]. For example, gpt-oss-20b, Qwen 3.6, and GLM 5.1 users reported issues after having this feature enabled by default.
• Grammar translation bugs: vLLM might have bugs in translating a model's tool schema or tool-calling format into the model's specific grammar. This can result in problems including invalid tool calls, request failures, silently dropping the constrained decoding, and lower tool-calling accuracy [1, 2].
• Feature interaction bugs: There are bugs in the interaction between constrained decoding and other serving-engine features. For example, a bug in the interplay between constrained decoding and speculative decoding can result in speculative tokens being wrongly rejected by the grammar, leading to failing requests [1, 2].

Due to these issues, vLLM maintainers reported seeing problems with several models after constrained decoding was enabled. This made disabling constrained decoding by default the safer option for a model-agnostic serving engine such as vLLM. Later, vLLM added the --tool-strict-level flag so that users can enable tool-call strictness for their deployments.

Ubicloud’s focus is different from that of vLLM however. We focus on serving Kimi and DeepSeek models with the quality that users expect from first-party providers. For this reason, we enabled constrained decoding and re-ran the K2VV ToolCall benchmark. Once we verified 100% tool-call schema accuracy across all served model versions, we deployed the change to production. We then typed up this blog post to share our learnings.

When using LLMs, the quality you observe isn’t based on the model alone. The inference engine also plays an important role. Change the inference engine and its settings, and you change the model’s output quality.

When serving Kimi K2.6 and DeepSeek V4 Flash, we observed visible quality degradations due to tool-call schema inaccuracies. We solved this problem by first carrying our own vLLM fork and then enabling the new constrained decoding feature in vLLM. This helped improve our tool-call accuracy in the K2VV ToolCall benchmark from 92% to 100%.

In this blog post, we shared our experience with tool calling and constrained decoding. If this is interesting or you’d like to share war stories about serving models, please drop us a line in our support email.