Original date published: June 7, 2026
AgenticML - Designing an action serialization interface format for autonomous LLM agents
Building AgenticML, a new markup language and serialization format for LLM agents as opposed to ChatML and Harmony
AgenticML is a new markup language and serialization format for representing agent tasks as robotic execution trajectories, as opposed to turn-based conversations. I developed a framework for designing new serialization formats for language agents and training agents to follow custom serialization formats. The results show that while the ChatML format performs better than AgenticML on standard benchmarks such as BFCL and ToolBench, the model is fully able to execute complex multi-turn operations using this format, despite the format not existing in the training data. While our format was more token- and latency-efficient, the performance challenge stems from the representation of the evaluation set as chat-based, turn-based conversations for execution; building new evaluation formats or systems that translate standard benchmarks to custom formats would be an interesting follow-up challenge. To achieve this, we had to customize the model itself from the base layer, from re-embedding reserved tokens to fit our new serialization format to retraining the model directly from the base model without any prior supervised fine-tuning or preference tuning. The work primarily uses Llama 3.1 8B base; we also explored using other models like Qwen 2.5 7B but ran into challenges accessing and customizing their reserved tokens. This work shows an exciting new direction for thinking about how to design agentic models and provides a foundation for moving away from turn-based conversational agents towards fully agentic runtime models.
Problem
The first era of language models was primarily driven by text completion; therefore, most language models were used through a completion interface, i.e., they functioned entirely within next-token prediction behaviour. As methods like RLHF and SFT developed, we were able to design models that behave like conversational assistants. This led to the release of ChatGPT as a chat-based assistant model, and OpenAI created the ChatCompletion interface for developers to build AI chat applications with it. This interface uses a serialization format known as ChatML (Harmony is a more recent version of ChatML but is still primarily built around turn-based conversation); in this interface, the interaction with the LLM is modelled as a chat session. The session is made up of messages of the following types:
- System
- User
- Assistant
- Developer (this is a newer message type implemented as part of ChatML)
The natural interface for this is a chatbox. It poses the interaction with AI systems as a conversation. Gradually, we added more modalities to the conversation, such as images and audio, but ultimately it is still based on turn-based conversations.
Today, we have moved on from working with AI models as chat assistants to fully autonomous workers. These are less like turn-based interactions and more like robotic mission planning and execution. Tools like Claude Code, Pi and other coding agents have made it possible to start imagining how such fully autonomous agent systems would behave, yet as agent developers, we are still interacting with AI models through an interface designed for conversations, not actions.
Chat Completion was designed for turn-taking dialogue, but agents do things that don't map cleanly to that, such as maintaining goals across long horizons, replanning as the world state changes, managing uncertainty about state, executing hierarchically (high-level intent -> mid-level plans -> low-level actions) and recovering from failures. While we can (and in many cases have) addressed these challenges within the chat interface format, we are simply working around a problem that does not need to exist anymore, which is hindering us from coming up with more innovative ideas around agentic execution.
Classical mobile robotics has decades of well-developed abstractions for those problems: the sense-plan-act loop, behaviour trees, hierarchical task networks (HTN), POMDPs, subsumption architectures, blackboard systems, ROS-style topic/service/action interfaces, and so on. The bet is that some of these would translate well into LM agents and give us a more principled interface than "stuff everything into a message list and hope."

Mobile Robotics Trajectory
Designing a new serialization interface for language agents
For the interface design itself, we are drawn to a combination of designing the runtime architecture and serialization formats (like ChatML but for agents); how do we design a token structure that would best fit agents?
Taking inspiration from robotics, we will be looking at the following methods to determine how best to design this new AgenticML interface:
- Behaviour trees / HTN, which are great for structured, hierarchical task decomposition with reactive fallback
- Sense-plan-act with belief state, which forces the model to maintain and update a world model and is good for partial observability
- ROS-style action interface with goal, feedback, result and preemption, which has clean async semantics
- MDP/POMDP framing, which has explicit states, actions, observations and rewards; this is the most principled but heaviest to implement
- Statecharts and OODA loops
Prior works have explored this problem but have not attempted to redesign the serialization interface itself. ReAct, Reflexion, Voyager, SayCan, Inner Monologue, Tree-of-Thought and behaviour-tree-structured prompting all focus on changing the message representation interface for agents, but ultimately the data passed to the LLMs is still represented as turn-based messages and therefore serialized to the model using the ChatML format.
This work does not focus on building agent communication protocols like ACP, MCP or A2A. These are outside the scope of what we are trying to build here, but it would be worth exploring how these protocols can be translated to custom formats.
Thesis and Challenges
ChatCompletion is a serialization format that was retrofitted for agents. We need to design a serialization format that is native to agency; I'm calling it AgenticML. We are training a base model from scratch to operate in this format from the pretraining/post-training boundary. This is not via SFT on top of an already instruction-tuned chat model. The interface borrows load-bearing concepts from ROS and the deliberative-reactive robotics literature. RL-friendliness is a property the interface is designed natively with.
Current chat-completion-as-agent collapses all layers into a single homogeneous stream. Every "turn" is the same kind of object, a message, regardless of whether it is a quick reaction, a sub-task delegation, or a full replan. That's serializing a sophisticated multi-rate control system as a flat sequence. The robotics community spent fifteen years discovering that this doesn't work and converging on layered designs. We're rediscovering this with LLMs.
For the interface design, we are currently exploring three layers: a protocol, a runtime architecture and a serialization method. Methods such as HBTP, BT-Gen and ReActTree all work by using BTs as an output format for LLM-based planners targeting actual robots. Nobody has proposed BTs as the native interface a general-purpose agent model is trained to operate in. The 1990 consensus in robotics was that reactive systems do not scale to long-horizon execution, yet the ReAct runtime architecture is still the most prevalent way to design autonomous agents, even in advanced systems like Claude Code or Pegasus. The robotics community decided that a hybrid architecture consisting of a deliberative planner at the top, a sequencer in the middle and reactive control at the bottom was the ideal architecture for designing robots for long-task execution.
Challenges:
- Have a full understanding of the existing serialization standard and runtime
- Design a new serialization standard and runtime
- Pick a base model to build the serialization format on top of
- Create a pipeline to test training a model with the existing data pipeline
- Create synthetic data for the new training pipeline
- Run the tuning pipeline with the new agent template
- Evaluate our format on BFCL and ToolBench against existing formats like ChatML
Understanding of the existing serialization standard and runtime
ChatML typically uses a list of dictionaries called messages, which form a conversation.
[
{
"role": "system",
"content": "You are a coding assistant.\n\ntools:\nnamespace tools {\n type read_file = (_: { path: string }) => any;\n type answer = (_: { text: string }) => any;\n}"
},
{"role": "user", "content": "How many lines are in main.py?"},
{
"role": "assistant",
"content": "",
"tool_calls": [{
"id": "call_2",
"type": "function",
"function": {"name": "read_file", "arguments": "{\"path\": \"main.py\"}"}
}]
},
{
"role": "tool",
"tool_call_id": "call_2",
"content": "{\"tool\": \"read_file\", \"value\": \"def main():\\n print('hi')\\n\"}"
},
{"role": "assistant", "content": "main.py has 2 lines."}
]
This is an example of an agentic task asking the model to read a file. This is typically what the developer passes to the model and the structure the developer builds on, but what the model sees is not this JSON object. Rather, it sees a wire sequence created by the tokenizer using the .apply_chat_template functionality. The wire format for this sequence would look like this:
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a coding assistant.
tools:
namespace tools {
type read_file = (_: { path: string }) => any;
type answer = (_: { text: string }) => any;
}<|eot_id|><|start_header_id|>user<|end_header_id|>
How many lines are in main.py?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
<|python_tag|>{"name": "read_file", "parameters": "{\"path\": \"main.py\"}"}<|eom_id|><|eot_id|><|start_header_id|>ipython<|end_header_id|>
{"tool": "read_file", "value": "def main():\n print('hi')\n"}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
main.py has 2 lines.<|eot_id|>
This is the format the base model sees. The ChatML format has a set of special tokens which the model is trained to respond and interact with. They guide the model's response and also the parsers that extract the generated output from the model. The special tokens include <|begin_of_text|>, <|start_header_id|>, <|eot_id|>, <|im_start|>, <|python_tag|>. These are the tokens that the model uses to guide its response.
This serialization format is designed primarily around turn-based conversation. In order to bypass the turn-based conversation pattern, we have to redesign our own serialization format as well.
Design a new serialization standard and runtime
A goal-native serialization format for language agents.
AgenticML is a research project exploring an alternative to chat-completion interfaces for autonomous agents. Where ChatML and Harmony treat every turn as a message with a role, AgenticML uses typed frames—goal, mission, belief, plan, think, action, result, feedback and reward—that are first-class objects in the serialization itself. Each frame is delimited by a single special token from the model's reserved vocabulary.
Rather than using conversations as in ChatML, we use trajectories, and rather than using messages, we use frames. This framing alone allows us to rethink how we interact with these models and what we expect to get from them.
Frame types
| Marker | Owner | Payload | Purpose |
|---|---|---|---|
<|goal|> | runtime | prose | Persistent role / objective. Always the first frame. |
<|mission|> | runtime | prose | Specific task instruction. Latest supersedes earlier. |
<|obs|> | runtime | prose | Runtime context (tool definitions, env, files, etc.). |
<|belief|> | model | prose | Persistent model-internal state. Updated each step. |
<|plan|> | model | prose | Strategy. Latest supersedes earlier. |
<|think|> | model | prose | Private reasoning. Stripped after consumption. |
<|action|> | model | JSON | Tool invocation. The model may emit multiple actions per turn. |
<|end|> | model | empty | Generation stop token. Model yields control to runtime. |
<|result|> | runtime | JSON | Tool outcome. One per action. |
<|feedback|> | runtime | prose | Tool progress or user follow-up. Persistent. |
<|reward|> | runtime | number | Training-time reward signal. Stripped after consumption. |
An example trajectory in AgenticML for the same file-reading task from earlier would look like this:
[
{"type": "goal", "content": "You are a coding assistant."},
{"type": "obs", "content": "tools:\nnamespace tools {\n type read_file = (_: { path: string }) => any;\n type answer = (_: { text: string }) => any;\n}"},
{"type": "mission", "content": "How many lines are in main.py?"},
{"type": "action", "content": {"tool": "read_file", "path": "main.py"}},
{"type": "end", "content": null},
{"type": "result", "content": {"tool": "read_file", "value": "def main():\n print('hi')\n"}},
{"type": "action", "content": {"tool": "answer", "text": "main.py has 2 lines."}},
{"type": "end", "content": null}
]
The wire format version of the sequence would look like this:
<|goal|>You are a coding assistant.<|mission|>How many lines are in main.py?<|obs|>tools:
namespace tools {
type read_file = (_: { path: string }) => any;
type answer = (_: { text: string }) => any;
}<|action|>{"tool":"read_file","path":"main.py"}<|end|><|result|>{"tool":"read_file","value":"def main():\n print('hi')\n"}<|belief|>main.py has 2 lines.<|action|>{"tool":"answer","text":"main.py has 2 lines."}<|end|>
We can already see that our format is more token efficient and is training the right primitives into the model.
Pick a base model to build the serialization format on top of
For the experiment I considered the following models:
-
Qwen 2.5-7B base: Apache 2.0, openly released as base (non-instruct) checkpoints, strong pretraining, well supported in the HF ecosystem and Unsloth
-
OLMo 2 13B base: Fully open, including training data, by AI2. Apache 2.0. Genuinely the most scientifically clean option because you can audit what went into pretraining. It is slightly weaker in terms of capabilities than other frontier open models, but the transparency is valuable for research projects where we would like to study behaviour from pretraining to fine-tuning.
-
Llama 3.1 8B base: This has a less permissive license but provides the additional reserved tokens we require to run the experiment. It is also a powerful base model for experimentation. The license limitations may make it difficult for commercial applications.
I initially decided to go with Qwen2.5-7B, but while working on the project, I realized that the Qwen models do not expose their reserved tokens for modification.
Qwen2.5 doesn't include a deliberate reserve. Compared to Llama-3, where Meta documented explicitly: "we added 250 reserved tokens to support more use-cases without requiring vocab resize." That was an intentional design decision. Qwen2.5 didn't make the same choice — its special tokens are all functional (ChatML markers, vision/grounding markers, endoftext). There's no pool of pre-allocated unused slots. AgenticML requires a base model with available reserved tokens, or willingness to resize the vocabulary.
OLMo is a good research model, but its performance was subpar for the project's requirements. I decided on Llama 3.1 8B base. It's a strong base model with a permissive license that is good enough for the project. Most importantly, it had enough reserved tokens that were open to modification.
Here is an example of the wire format used in the instruction-tuned version of Llama 3.1.
Example format for Llama 3.1
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>
What is 2+2?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
4<|eot_id|>
We will be training the model to use our own AgenticML format as opposed to this default format.
Design decisions
- Base model: Llama-3.1-8B-base. Llama-3 has 250 documented reserved
special tokens; Qwen2.5 does not. AgenticML claims eleven of these
reserved slots and aliases them to frame markers at the string level:
AgenticML marker Reserved token Token ID <|goal|><|reserved_special_token_0|>128002 <|mission|><|reserved_special_token_1|>128003 <|obs|><|reserved_special_token_2|>128005 <|belief|><|reserved_special_token_3|>128011 <|plan|><|reserved_special_token_4|>128012 <|think|><|reserved_special_token_5|>128013 <|action|><|reserved_special_token_6|>128014 <|end|><|reserved_special_token_7|>128015 <|result|><|reserved_special_token_8|>128016 <|feedback|><|reserved_special_token_9|>128017 <|reward|><|reserved_special_token_10|>128018 - No closing tokens. Each frame extends until the next marker or
<|end|>. <|end|>is a stored frame and generation stop. Each model turn ends with an explicitendframe in the trajectory; the model emits<|end|>to yield control to the runtime.- Batched actions per turn. A model block may emit multiple actions; the runtime emits a result for each, in order.
- No action IDs in v1. Action-result correspondence is positional.
- TypeScript-namespace tool schemas (Harmony-style). The model has seen this format in pretraining.
- Strict ownership at the parser. Model output cannot contain runtime-owned frames in strict mode.
- Tool schemas are caller-owned. Use
with_tool_obs(trajectory, tools)once beforestep();step()tokenizes the trajectory as given.
Create a pipeline to test training a model with the existing data format
To test the validity of our approach, we would test both a base model trained on ChatML and a base model trained on AgenticML. This would give us a platform for a fair comparison between the two approaches.
Before training the base model, we would have to initialize the reserved tokens because they were not used at all during training. We perform two primary activities during the initialization phase. First, we update the tokenizer to use our template. Second, we update the input and output embeddings of the decoder model to be more familiar with our format by applying the mean of internal representations of related words as the starting-point embedding for our model. We do this for both the ChatML and AgenticML formats.
# map each agenticml marker to seed token strings to average.
AGENTICML_SEED_TOKENS: dict[str, list[str]] = {
"<|goal|>": ["goal", "objective", "purpose", "aim"],
"<|mission|>": ["mission", "task", "instruction", "assignment", "problem"],
"<|obs|>": ["observation", "context", "environment", "situation"],
"<|belief|>": ["belief", "state", "knowledge", "assumption"],
"<|plan|>": ["plan", "strategy", "approach", "method"],
"<|think|>": ["think", "reasoning", "thought", "reflection"],
"<|action|>": ["action", "call", "tool", "command", "invocation", "function"],
"<|end|>": ["end", "stop", "done", "complete", "finish", "terminate"],
"<|result|>": ["result", "output", "response", "outcome"],
"<|feedback|>": ["feedback", "update", "progress", "comment"],
"<|reward|>": ["reward", "score", "bonus", "credit"],
}
AGENTICML_MARKER_SLOT: dict[str, int] = {
frame_type.value: slot for frame_type, slot in AGENTICML_TOKEN_MAP
}
def _seed_token_ids(tokenizer, words: list[str]) -> list[int]:
"""encode seed words and return single-token ids only."""
out: list[int] = []
for w in words:
ids = tokenizer.encode(" " + w, add_special_tokens=False)
if len(ids) == 1:
out.append(ids[0])
else:
print(f" warning: {w!r} tokenized to {len(ids)} tokens, skipping")
if not out:
first = tokenizer.encode(" " + words[0], add_special_tokens=False)
out = [first[0]]
print(" fallback: using first sub-word token only")
return out
def initialize_agenticml_embeddings(model, tokenizer) -> None:
"""in-place modification of embed_tokens and lm_head rows."""
if model.config.tie_word_embeddings:
raise RuntimeError(
"this script assumes untied embeddings; llama-3.1 has tie_word_embeddings=False. "
"got tie_word_embeddings=True - the script would need a different code path."
)
embed = model.get_input_embeddings().weight
lm_head = model.get_output_embeddings().weight
reserved_slot_base = require_single_token_id(tokenizer, "<|reserved_special_token_0|>")
print(f"embed_tokens device: {embed.device}, dtype: {embed.dtype}")
print(f"lm_head device: {lm_head.device}, dtype: {lm_head.dtype}")
print(f"reserved_slot_0 base id: {reserved_slot_base}")
for marker, seeds in AGENTICML_SEED_TOKENS.items():
slot = AGENTICML_MARKER_SLOT[marker]
reserved_name = f"<|reserved_special_token_{slot}|>"
target_id = require_single_token_id(tokenizer, reserved_name)
print(f"\n{marker} -> {reserved_name} (id={target_id})")
seed_ids = _seed_token_ids(tokenizer, seeds)
print(f" seed token ids: {seed_ids}")
seed_embed = embed[seed_ids].float().mean(dim=0)
seed_head = lm_head[seed_ids].float().mean(dim=0)
with torch.no_grad():
embed[target_id] = seed_embed.to(embed.dtype)
lm_head[target_id] = seed_head.to(lm_head.dtype)
new_norm_embed = embed[target_id].float().norm().item()
new_norm_head = lm_head[target_id].float().norm().item()
print(f" embed norm: {new_norm_embed:.4f}")
print(f" lm_head norm: {new_norm_head:.4f}")
def run_initialize_agenticml_embeddings(
base_model_id: str,
*,
output_dir: str | Path,
repo_id: str | None = None,
) -> None:
# tokenizer only: vocab ids for reserved slots and seed words (vocab is unchanged).
print(f"loading tokenizer: {base_model_id}")
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
# model weights: embed_tokens / lm_head rows are updated in place.
print(f"loading model weights: {base_model_id}")
model = AutoModelForCausalLM.from_pretrained(
base_model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
)
print("\ninitializing agenticml marker embeddings...")
initialize_agenticml_embeddings(model, tokenizer)
print("\nattaching agenticml template to tokenizer...")
bake_agentic_template(tokenizer)
out = Path(output_dir)
print(f"\nsaving to {out}")
model.save_pretrained(out)
tokenizer.save_pretrained(out)
if not repo_id:
print("done (no --repo-id; skipped hub push).")
return
print(f"\npushing to huggingface hub: {repo_id}")
commit_message = f"agenticml embedding init from base {base_model_id}"
model.push_to_hub(repo_id, commit_message=commit_message) # type: ignore[call-arg]
tokenizer.push_to_hub(repo_id, commit_message=commit_message) # type: ignore[call-arg]
print("pushed.")
print("done.")
Once the initialization is complete, we push the initialized versions of the models to the Hub. These are the base versions we will train on top of. You can find the initialized versions of the Llama 3.1 8B models I created for AgenticML and ChatML on Hugging Face.
Create synthetic data for the new training pipeline
The entire training process is done on synthetic data. I created a synthetic data generation pipeline that generates accurate agentic trajectories using Qwen 3.5 Plus. The trajectories are generated in the following domains:
| Domain | Share | Slug |
|---|---|---|
| Coding and dev tools | 12% | coding |
| Personal finance | 8% | finance |
| Scheduling and calendar | 6% | scheduling |
| Cooking and meal planning | 6% | cooking |
| Research and fact synthesis | 8% | research |
| News and current events | 4% | news |
| Travel planning | 6% | travel |
| Writing and communication | 8% | writing |
| Tutoring and education | 6% | education |
| Health information | 4% | health |
| Home and household ops | 4% | home |
| Smart home / IoT | 4% | smart_home |
| Shopping and gifts | 4% | shopping |
| Music and media curation | 4% | media |
| Photo and file curation | 3% | photo |
| Interpersonal advice | 3% | interpersonal |
| Legal and contract review | 3% | legal |
| Vehicle / repair diagnosis | 2% | vehicle |
| Civic / non-profit / charity | 3% | civic |
| Customer support | 2% | support |
We use a crafted prompt and a parallel generation pipeline to generate the dataset in the AgenticML format. We then use a follow-on script to convert the trajectories into ChatML, which is used to train the base ChatML model for comparison. Once completed, I pushed the training data to the Hugging Face Hub, where it can serve as a foundation for future work. The dataset can be found here.
Run the training pipeline with the new agent template
For training, here is how we set up our experiment configuration and objectives:
Experimental Setup
| Item | Value |
|---|---|
| Base | Llama 3.1 8B |
| Data | kosiasuzu/agenticml-agent-trajectory-dataset (frames / messages paired) |
| Init | Reserved-slot embeddings (AgenticML) vs instruct tokenizer rows (ChatML) |
| Training | LoRA r=32, α=64, 2 epochs, cosine LR 2e-4, batch 1 × grad accum 32 |
| Models | agenticml-llama3.1-8b-lora-merged, chatml-llama3.1-8b-lora-merged |
| Pod | RunPod w7ldz9cztwcljf, NVIDIA L40S |
| Command | agenticml eval-run-all --suites bfcl toolbench format_validity --continue-on-error |
| BFCL subset | 45 pinned IDs in src/agenticml/evaluation/benchmarks/bfcl/subset.py, seed 42 |
| ToolBench subset | 10 pinned G1_instruction query IDs in toolbench/subset.py |
Research Questions
- RQ1 — Format Validity: Can each format produce parseable, structurally valid agent output?
- RQ2 — Downstream benchmarks: Does serialization affect BFCL / ToolBench under matched SFT?
- RQ3 — Efficiency: Token and latency cost per successful BFCL task?
Here is a code snippet of the training configurations we used for the LoRA training of both models.
def build_lora_model(model_id: str):
model = AutoModelForCausalLM.from_pretrained(
model_id,
dtype=torch.bfloat16,
)
model.gradient_checkpointing_enable()
model.enable_input_require_grads()
lora_cfg = LoraConfig(
r=32,
lora_alpha=64,
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
target_modules=[
"q_proj",
"k_proj",
"v_proj",
"o_proj",
"gate_proj",
"up_proj",
"down_proj",
"lm_head",
],
)
return get_peft_model(model, lora_cfg)
def build_full_model(model_id: str):
model = AutoModelForCausalLM.from_pretrained(
model_id,
dtype=torch.bfloat16,
)
model.gradient_checkpointing_enable()
return model
def build_model(model_id: str, mode: TrainingMode):
if mode is TrainingMode.LORA:
return build_lora_model(model_id)
return build_full_model(model_id)
def print_trainable(model) -> None:
trainable = 0
total = 0
for p in model.parameters():
total += p.numel()
if p.requires_grad:
trainable += p.numel()
pct = 100 * trainable / total
print(f"trainable params: {trainable:,} / {total:,} ({pct:.2f}%)")
We trained both the AgenticML model and the ChatML model on the full datasets, and each took about 2.5 hours on an L40S GPU.

AgenticML & ChatML Training

AgenticML & ChatML Evaluations
Once the models were trained, I pushed both to the Hugging Face Hub: the AgenticML model and the ChatML model. We also deployed the LoRA adapters here and here.
Evaluating our format on BFCL and ToolBench against existing formats like ChatML
We built evaluation sets on top of ToolBench, Berkeley Function Calling Leaderboard (BFCL) and SWEBench-Mini. These benchmarks were chosen because they are representative benchmarks of agentic tasks. I had to drop SWEBench-Mini due to the large computational requirements needed to run it, so I focused on ToolBench and BFCL. I had to write custom bridges for the evaluations and scoring algorithms to map the ChatML format used by the evaluation benchmarks to the AgenticML format. We also created a format-validity benchmark to ensure the trained models were outputting responses in the correct format. Here are the results of the experiments.
| Suite | Format | Model | n | Primary | Secondary | Avg Tokens | Tok/Success | Avg Wall Sec |
|---|---|---|---|---|---|---|---|---|
| BFCL | AgenticML | agenticml-llama3.1-8b-lora-merged | 45 | 4.4% (accuracy) | 3.9 (avg_retry_count) | 140,018 | 1,363 | 223.9 |
| BFCL | ChatML | chatml-llama3.1-8b-lora-merged | 45 | 60.0% (accuracy) | 1.6 (avg_retry_count) | 14,268 | 3,775 | 3.9 |
| ToolBench | AgenticML | agenticml-llama3.1-8b-lora-merged | 10 | 0.0% (pass_rate) | 1.0 (avg_steps) | 2,121 | — | 29.7 |
| ToolBench | ChatML | chatml-llama3.1-8b-lora-merged | 10 | 0.0% (pass_rate) | 9.8 (avg_steps) | 21,542 | — | 11.4 |
| Format Validity | AgenticML | agenticml-llama3.1-8b-lora-merged | smoke | 100% (valid_rate) | 100% (parse_rate) | — | — | — |
| Format Validity | ChatML | chatml-llama3.1-8b-lora-merged | smoke | 100% (valid_rate) | 100% (parse_rate) | — | — | — |

Suite Primary Metrics
Here is the breakdown of BFCL by category:
| Category | AgenticML | ChatML |
|---|---|---|
| simple_python | 0/12 | 12/12 |
| multiple | 0/4 | 4/4 |
| parallel | 0/4 | 0/4 |
| parallel_multiple | 2/4 | 0/4 |
| live_simple | 0/7 | 4/7 |
| live_multiple | 0/6 | 6/6 |
| multi_turn_base | 0/8 | 1/8 |
Here is a breakdown of the failures in BFCL:
43 failed tasks:
| Shape | Count | Meaning |
|---|---|---|
| json_tool_call | 28 | Model emitted tool JSON; BFCL grader still rejected |
| multi_turn_all_empty | 8 | No usable turn output across multi-turn loop |
| empty_no_tool_call | 7 | Encoded result was [] |
This is a breakdown of the failure modes for AgenticML in the evaluation:
- Duplicate parallel arrays on single-call tasks — e.g.
simple_python_52,_57,_71,_114. Ground truth expects one{"name": ..., "parameters": ...}. Multiple action frames in one step →actions_to_resultbuilds a JSON array. ChatML emits a single FC object → 12/12 on simple_python. - Multi-turn collapse (0/8) — Loop stops when a step returns
[]or parse error. - Live schema mismatch (0/13 AgenticML vs 10/13 ChatML) — Live categories need exact dotted tool names; ChatML uses BFCL’s native system prompt; AgenticML uses goal/mission + TypeScript
with_tool_obs. - Parallel categories — ChatML often emits one call when several are required; AgenticML sometimes over-calls.
parallel_multiple2/4 is the one category where AgenticML beat ChatML.

BFCL Failure per Category
All 10 ToolBench tasks completed inference and scoring. Every row failed structurally with no finish (no terminal finish action within step budget). Infra is fine; models need better multi-step tool use on G1_instruction.
AgenticML BFCL runs averaged 140k tokens and 224s wall time per task vs ChatML 14k tokens and 3.9s — largely driven by retry loops and multi-turn empties, not raw inference speed alone.

BFCL Efficiency Comparison
Conclusion
While I did not get the results I was expecting, I think this serves as a foundation for a new way of thinking about how we build agentic models. With more compute resources, I would be able to pursue some of these gaps in the performance of the AgenticML format, explore how to build custom evaluations for AgenticML and convert existing evaluations in ChatML into the AgenticML format for fairer evaluations. Here are some brief AI-generated takeaways from the work so far:
Prompt alignment vs BFCL: ChatML eval uses BFCL’s native ChatML + function-call path. AgenticML uses our frame runtime and bridge into gorilla result JSON. That asymmetry is intentional (each format evaluated in its natural harness) but it explains much of the live-category gap.
Eval fairness: Treating AgenticML 4.4% as pure capability understates the format. A harness fix (single-call collapse for non-parallel categories) is the highest-leverage ablation before retraining.
When AgenticML may still help: Trajectory clarity, custom runtimes, SWE-style long horizons, domains where ChatML FC is not the target wire format.
Limitations: 8B only; synthetic data; Format Validity not at n=100 on RunPod; SWE-bench skipped; no harness-fix ablation yet; ToolBench 0% both formats.
The entire code repository for the work can be found here: https://github.com/asuzukosi/agenticml
Last updated at: June 7, 2026