Original date published: June 7, 2026

AgenticML - Designing an action serialization interface format for autonomous LLM agents

Building AgenticML, a new markup language and serialization format for LLM agents as opposed to ChatML and Harmony


AgenticML is a new markup language and serialization format for representing agent tasks as robotic execution trajectories, as opposed to turn-based conversations. I developed a framework for designing new serialization formats for language agents and training agents to follow custom serialization formats. The results show that while the ChatML format performs better than AgenticML on standard benchmarks such as BFCL and ToolBench, the model is fully able to execute complex multi-turn operations using this format, despite the format not existing in the training data. While our format was more token- and latency-efficient, the performance challenge stems from the representation of the evaluation set as chat-based, turn-based conversations for execution; building new evaluation formats or systems that translate standard benchmarks to custom formats would be an interesting follow-up challenge. To achieve this, we had to customize the model itself from the base layer, from re-embedding reserved tokens to fit our new serialization format to retraining the model directly from the base model without any prior supervised fine-tuning or preference tuning. The work primarily uses Llama 3.1 8B base; we also explored using other models like Qwen 2.5 7B but ran into challenges accessing and customizing their reserved tokens. This work shows an exciting new direction for thinking about how to design agentic models and provides a foundation for moving away from turn-based conversational agents towards fully agentic runtime models.

Problem

The first era of language models was primarily driven by text completion; therefore, most language models were used through a completion interface, i.e., they functioned entirely within next-token prediction behaviour. As methods like RLHF and SFT developed, we were able to design models that behave like conversational assistants. This led to the release of ChatGPT as a chat-based assistant model, and OpenAI created the ChatCompletion interface for developers to build AI chat applications with it. This interface uses a serialization format known as ChatML (Harmony is a more recent version of ChatML but is still primarily built around turn-based conversation); in this interface, the interaction with the LLM is modelled as a chat session. The session is made up of messages of the following types:

  • System
  • User
  • Assistant
  • Developer (this is a newer message type implemented as part of ChatML)

The natural interface for this is a chatbox. It poses the interaction with AI systems as a conversation. Gradually, we added more modalities to the conversation, such as images and audio, but ultimately it is still based on turn-based conversations.

Today, we have moved on from working with AI models as chat assistants to fully autonomous workers. These are less like turn-based interactions and more like robotic mission planning and execution. Tools like Claude Code, Pi and other coding agents have made it possible to start imagining how such fully autonomous agent systems would behave, yet as agent developers, we are still interacting with AI models through an interface designed for conversations, not actions.

Chat Completion was designed for turn-taking dialogue, but agents do things that don't map cleanly to that, such as maintaining goals across long horizons, replanning as the world state changes, managing uncertainty about state, executing hierarchically (high-level intent -> mid-level plans -> low-level actions) and recovering from failures. While we can (and in many cases have) addressed these challenges within the chat interface format, we are simply working around a problem that does not need to exist anymore, which is hindering us from coming up with more innovative ideas around agentic execution.

Classical mobile robotics has decades of well-developed abstractions for those problems: the sense-plan-act loop, behaviour trees, hierarchical task networks (HTN), POMDPs, subsumption architectures, blackboard systems, ROS-style topic/service/action interfaces, and so on. The bet is that some of these would translate well into LM agents and give us a more principled interface than "stuff everything into a message list and hope."

Mobile Robotics Trajectory

Mobile Robotics Trajectory

Designing a new serialization interface for language agents

For the interface design itself, we are drawn to a combination of designing the runtime architecture and serialization formats (like ChatML but for agents); how do we design a token structure that would best fit agents?

Taking inspiration from robotics, we will be looking at the following methods to determine how best to design this new AgenticML interface:

  • Behaviour trees / HTN, which are great for structured, hierarchical task decomposition with reactive fallback
  • Sense-plan-act with belief state, which forces the model to maintain and update a world model and is good for partial observability
  • ROS-style action interface with goal, feedback, result and preemption, which has clean async semantics
  • MDP/POMDP framing, which has explicit states, actions, observations and rewards; this is the most principled but heaviest to implement
  • Statecharts and OODA loops

Prior works have explored this problem but have not attempted to redesign the serialization interface itself. ReAct, Reflexion, Voyager, SayCan, Inner Monologue, Tree-of-Thought and behaviour-tree-structured prompting all focus on changing the message representation interface for agents, but ultimately the data passed to the LLMs is still represented as turn-based messages and therefore serialized to the model using the ChatML format.

This work does not focus on building agent communication protocols like ACP, MCP or A2A. These are outside the scope of what we are trying to build here, but it would be worth exploring how these protocols can be translated to custom formats.

Thesis and Challenges

ChatCompletion is a serialization format that was retrofitted for agents. We need to design a serialization format that is native to agency; I'm calling it AgenticML. We are training a base model from scratch to operate in this format from the pretraining/post-training boundary. This is not via SFT on top of an already instruction-tuned chat model. The interface borrows load-bearing concepts from ROS and the deliberative-reactive robotics literature. RL-friendliness is a property the interface is designed natively with.

Current chat-completion-as-agent collapses all layers into a single homogeneous stream. Every "turn" is the same kind of object, a message, regardless of whether it is a quick reaction, a sub-task delegation, or a full replan. That's serializing a sophisticated multi-rate control system as a flat sequence. The robotics community spent fifteen years discovering that this doesn't work and converging on layered designs. We're rediscovering this with LLMs.

For the interface design, we are currently exploring three layers: a protocol, a runtime architecture and a serialization method. Methods such as HBTP, BT-Gen and ReActTree all work by using BTs as an output format for LLM-based planners targeting actual robots. Nobody has proposed BTs as the native interface a general-purpose agent model is trained to operate in. The 1990 consensus in robotics was that reactive systems do not scale to long-horizon execution, yet the ReAct runtime architecture is still the most prevalent way to design autonomous agents, even in advanced systems like Claude Code or Pegasus. The robotics community decided that a hybrid architecture consisting of a deliberative planner at the top, a sequencer in the middle and reactive control at the bottom was the ideal architecture for designing robots for long-task execution.

Challenges:

  • Have a full understanding of the existing serialization standard and runtime
  • Design a new serialization standard and runtime
  • Pick a base model to build the serialization format on top of
  • Create a pipeline to test training a model with the existing data pipeline
  • Create synthetic data for the new training pipeline
  • Run the tuning pipeline with the new agent template
  • Evaluate our format on BFCL and ToolBench against existing formats like ChatML

Understanding of the existing serialization standard and runtime

ChatML typically uses a list of dictionaries called messages, which form a conversation.

[
  {
    "role": "system",
    "content": "You are a coding assistant.\n\ntools:\nnamespace tools {\n  type read_file = (_: { path: string }) => any;\n  type answer = (_: { text: string }) => any;\n}"
  },
  {"role": "user", "content": "How many lines are in main.py?"},
  {
    "role": "assistant",
    "content": "",
    "tool_calls": [{
      "id": "call_2",
      "type": "function",
      "function": {"name": "read_file", "arguments": "{\"path\": \"main.py\"}"}
    }]
  },
  {
    "role": "tool",
    "tool_call_id": "call_2",
    "content": "{\"tool\": \"read_file\", \"value\": \"def main():\\n    print('hi')\\n\"}"
  },
  {"role": "assistant", "content": "main.py has 2 lines."}
]

This is an example of an agentic task asking the model to read a file. This is typically what the developer passes to the model and the structure the developer builds on, but what the model sees is not this JSON object. Rather, it sees a wire sequence created by the tokenizer using the .apply_chat_template functionality. The wire format for this sequence would look like this:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a coding assistant.

tools:
namespace tools {
  type read_file = (_: { path: string }) => any;
  type answer = (_: { text: string }) => any;
}<|eot_id|><|start_header_id|>user<|end_header_id|>

How many lines are in main.py?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

<|python_tag|>{"name": "read_file", "parameters": "{\"path\": \"main.py\"}"}<|eom_id|><|eot_id|><|start_header_id|>ipython<|end_header_id|>

{"tool": "read_file", "value": "def main():\n    print('hi')\n"}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

main.py has 2 lines.<|eot_id|>

This is the format the base model sees. The ChatML format has a set of special tokens which the model is trained to respond and interact with. They guide the model's response and also the parsers that extract the generated output from the model. The special tokens include <|begin_of_text|>, <|start_header_id|>, <|eot_id|>, <|im_start|>, <|python_tag|>. These are the tokens that the model uses to guide its response.

This serialization format is designed primarily around turn-based conversation. In order to bypass the turn-based conversation pattern, we have to redesign our own serialization format as well.

Design a new serialization standard and runtime

A goal-native serialization format for language agents.

AgenticML is a research project exploring an alternative to chat-completion interfaces for autonomous agents. Where ChatML and Harmony treat every turn as a message with a role, AgenticML uses typed frames—goal, mission, belief, plan, think, action, result, feedback and reward—that are first-class objects in the serialization itself. Each frame is delimited by a single special token from the model's reserved vocabulary.

Rather than using conversations as in ChatML, we use trajectories, and rather than using messages, we use frames. This framing alone allows us to rethink how we interact with these models and what we expect to get from them.

Frame types

MarkerOwnerPayloadPurpose
<|goal|>runtimeprosePersistent role / objective. Always the first frame.
<|mission|>runtimeproseSpecific task instruction. Latest supersedes earlier.
<|obs|>runtimeproseRuntime context (tool definitions, env, files, etc.).
<|belief|>modelprosePersistent model-internal state. Updated each step.
<|plan|>modelproseStrategy. Latest supersedes earlier.
<|think|>modelprosePrivate reasoning. Stripped after consumption.
<|action|>modelJSONTool invocation. The model may emit multiple actions per turn.
<|end|>modelemptyGeneration stop token. Model yields control to runtime.
<|result|>runtimeJSONTool outcome. One per action.
<|feedback|>runtimeproseTool progress or user follow-up. Persistent.
<|reward|>runtimenumberTraining-time reward signal. Stripped after consumption.

An example trajectory in AgenticML for the same file-reading task from earlier would look like this:

[
  {"type": "goal", "content": "You are a coding assistant."},
  {"type": "obs", "content": "tools:\nnamespace tools {\n  type read_file = (_: { path: string }) => any;\n  type answer = (_: { text: string }) => any;\n}"},
  {"type": "mission", "content": "How many lines are in main.py?"},
  {"type": "action", "content": {"tool": "read_file", "path": "main.py"}},
  {"type": "end", "content": null},
  {"type": "result", "content": {"tool": "read_file", "value": "def main():\n    print('hi')\n"}},
  {"type": "action", "content": {"tool": "answer", "text": "main.py has 2 lines."}},
  {"type": "end", "content": null}
]

The wire format version of the sequence would look like this:

<|goal|>You are a coding assistant.<|mission|>How many lines are in main.py?<|obs|>tools:
namespace tools {
  type read_file = (_: { path: string }) => any;
  type answer = (_: { text: string }) => any;
}<|action|>{"tool":"read_file","path":"main.py"}<|end|><|result|>{"tool":"read_file","value":"def main():\n    print('hi')\n"}<|belief|>main.py has 2 lines.<|action|>{"tool":"answer","text":"main.py has 2 lines."}<|end|>

We can already see that our format is more token efficient and is training the right primitives into the model.

Pick a base model to build the serialization format on top of

For the experiment I considered the following models:

  • Qwen 2.5-7B base: Apache 2.0, openly released as base (non-instruct) checkpoints, strong pretraining, well supported in the HF ecosystem and Unsloth

  • OLMo 2 13B base: Fully open, including training data, by AI2. Apache 2.0. Genuinely the most scientifically clean option because you can audit what went into pretraining. It is slightly weaker in terms of capabilities than other frontier open models, but the transparency is valuable for research projects where we would like to study behaviour from pretraining to fine-tuning.

  • Llama 3.1 8B base: This has a less permissive license but provides the additional reserved tokens we require to run the experiment. It is also a powerful base model for experimentation. The license limitations may make it difficult for commercial applications.

I initially decided to go with Qwen2.5-7B, but while working on the project, I realized that the Qwen models do not expose their reserved tokens for modification.

Qwen2.5 doesn't include a deliberate reserve. Compared to Llama-3, where Meta documented explicitly: "we added 250 reserved tokens to support more use-cases without requiring vocab resize." That was an intentional design decision. Qwen2.5 didn't make the same choice — its special tokens are all functional (ChatML markers, vision/grounding markers, endoftext). There's no pool of pre-allocated unused slots. AgenticML requires a base model with available reserved tokens, or willingness to resize the vocabulary.

OLMo is a good research model, but its performance was subpar for the project's requirements. I decided on Llama 3.1 8B base. It's a strong base model with a permissive license that is good enough for the project. Most importantly, it had enough reserved tokens that were open to modification.

Here is an example of the wire format used in the instruction-tuned version of Llama 3.1.

Example format for Llama 3.1

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>

What is 2+2?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

4<|eot_id|>

We will be training the model to use our own AgenticML format as opposed to this default format.

Design decisions

  • Base model: Llama-3.1-8B-base. Llama-3 has 250 documented reserved special tokens; Qwen2.5 does not. AgenticML claims eleven of these reserved slots and aliases them to frame markers at the string level:
    AgenticML markerReserved tokenToken ID
    <|goal|><|reserved_special_token_0|>128002
    <|mission|><|reserved_special_token_1|>128003
    <|obs|><|reserved_special_token_2|>128005
    <|belief|><|reserved_special_token_3|>128011
    <|plan|><|reserved_special_token_4|>128012
    <|think|><|reserved_special_token_5|>128013
    <|action|><|reserved_special_token_6|>128014
    <|end|><|reserved_special_token_7|>128015
    <|result|><|reserved_special_token_8|>128016
    <|feedback|><|reserved_special_token_9|>128017
    <|reward|><|reserved_special_token_10|>128018
  • No closing tokens. Each frame extends until the next marker or <|end|>.
  • <|end|> is a stored frame and generation stop. Each model turn ends with an explicit end frame in the trajectory; the model emits <|end|> to yield control to the runtime.
  • Batched actions per turn. A model block may emit multiple actions; the runtime emits a result for each, in order.
  • No action IDs in v1. Action-result correspondence is positional.
  • TypeScript-namespace tool schemas (Harmony-style). The model has seen this format in pretraining.
  • Strict ownership at the parser. Model output cannot contain runtime-owned frames in strict mode.
  • Tool schemas are caller-owned. Use with_tool_obs(trajectory, tools) once before step(); step() tokenizes the trajectory as given.

Create a pipeline to test training a model with the existing data format

To test the validity of our approach, we would test both a base model trained on ChatML and a base model trained on AgenticML. This would give us a platform for a fair comparison between the two approaches.

Before training the base model, we would have to initialize the reserved tokens because they were not used at all during training. We perform two primary activities during the initialization phase. First, we update the tokenizer to use our template. Second, we update the input and output embeddings of the decoder model to be more familiar with our format by applying the mean of internal representations of related words as the starting-point embedding for our model. We do this for both the ChatML and AgenticML formats.

# map each agenticml marker to seed token strings to average.
AGENTICML_SEED_TOKENS: dict[str, list[str]] = {
    "<|goal|>": ["goal", "objective", "purpose", "aim"],
    "<|mission|>": ["mission", "task", "instruction", "assignment", "problem"],
    "<|obs|>": ["observation", "context", "environment", "situation"],
    "<|belief|>": ["belief", "state", "knowledge", "assumption"],
    "<|plan|>": ["plan", "strategy", "approach", "method"],
    "<|think|>": ["think", "reasoning", "thought", "reflection"],
    "<|action|>": ["action", "call", "tool", "command", "invocation", "function"],
    "<|end|>": ["end", "stop", "done", "complete", "finish", "terminate"],
    "<|result|>": ["result", "output", "response", "outcome"],
    "<|feedback|>": ["feedback", "update", "progress", "comment"],
    "<|reward|>": ["reward", "score", "bonus", "credit"],
}

AGENTICML_MARKER_SLOT: dict[str, int] = {
    frame_type.value: slot for frame_type, slot in AGENTICML_TOKEN_MAP
}


def _seed_token_ids(tokenizer, words: list[str]) -> list[int]:
    """encode seed words and return single-token ids only."""
    out: list[int] = []
    for w in words:
        ids = tokenizer.encode(" " + w, add_special_tokens=False)
        if len(ids) == 1:
            out.append(ids[0])
        else:
            print(f"  warning: {w!r} tokenized to {len(ids)} tokens, skipping")
    if not out:
        first = tokenizer.encode(" " + words[0], add_special_tokens=False)
        out = [first[0]]
        print("  fallback: using first sub-word token only")
    return out


def initialize_agenticml_embeddings(model, tokenizer) -> None:
    """in-place modification of embed_tokens and lm_head rows."""
    if model.config.tie_word_embeddings:
        raise RuntimeError(
            "this script assumes untied embeddings; llama-3.1 has tie_word_embeddings=False. "
            "got tie_word_embeddings=True - the script would need a different code path."
        )

    embed = model.get_input_embeddings().weight
    lm_head = model.get_output_embeddings().weight
    reserved_slot_base = require_single_token_id(tokenizer, "<|reserved_special_token_0|>")

    print(f"embed_tokens device: {embed.device}, dtype: {embed.dtype}")
    print(f"lm_head device:      {lm_head.device}, dtype: {lm_head.dtype}")
    print(f"reserved_slot_0 base id: {reserved_slot_base}")

    for marker, seeds in AGENTICML_SEED_TOKENS.items():
        slot = AGENTICML_MARKER_SLOT[marker]
        reserved_name = f"<|reserved_special_token_{slot}|>"
        target_id = require_single_token_id(tokenizer, reserved_name)
        print(f"\n{marker} -> {reserved_name} (id={target_id})")

        seed_ids = _seed_token_ids(tokenizer, seeds)
        print(f"  seed token ids: {seed_ids}")

        seed_embed = embed[seed_ids].float().mean(dim=0)
        seed_head = lm_head[seed_ids].float().mean(dim=0)

        with torch.no_grad():
            embed[target_id] = seed_embed.to(embed.dtype)
            lm_head[target_id] = seed_head.to(lm_head.dtype)

        new_norm_embed = embed[target_id].float().norm().item()
        new_norm_head = lm_head[target_id].float().norm().item()
        print(f"  embed norm: {new_norm_embed:.4f}")
        print(f"  lm_head norm: {new_norm_head:.4f}")


def run_initialize_agenticml_embeddings(
    base_model_id: str,
    *,
    output_dir: str | Path,
    repo_id: str | None = None,
) -> None:
    # tokenizer only: vocab ids for reserved slots and seed words (vocab is unchanged).
    print(f"loading tokenizer: {base_model_id}")
    tokenizer = AutoTokenizer.from_pretrained(base_model_id)

    # model weights: embed_tokens / lm_head rows are updated in place.
    print(f"loading model weights: {base_model_id}")
    model = AutoModelForCausalLM.from_pretrained(
        base_model_id,
        torch_dtype=torch.bfloat16,
        device_map="auto",
    )

    print("\ninitializing agenticml marker embeddings...")
    initialize_agenticml_embeddings(model, tokenizer)

    print("\nattaching agenticml template to tokenizer...")
    bake_agentic_template(tokenizer)

    out = Path(output_dir)
    print(f"\nsaving to {out}")
    model.save_pretrained(out)
    tokenizer.save_pretrained(out)

    if not repo_id:
        print("done (no --repo-id; skipped hub push).")
        return

    print(f"\npushing to huggingface hub: {repo_id}")
    commit_message = f"agenticml embedding init from base {base_model_id}"
    model.push_to_hub(repo_id, commit_message=commit_message)  # type: ignore[call-arg]
    tokenizer.push_to_hub(repo_id, commit_message=commit_message)  # type: ignore[call-arg]
    print("pushed.")
    print("done.")

Once the initialization is complete, we push the initialized versions of the models to the Hub. These are the base versions we will train on top of. You can find the initialized versions of the Llama 3.1 8B models I created for AgenticML and ChatML on Hugging Face.

Create synthetic data for the new training pipeline

The entire training process is done on synthetic data. I created a synthetic data generation pipeline that generates accurate agentic trajectories using Qwen 3.5 Plus. The trajectories are generated in the following domains:

DomainShareSlug
Coding and dev tools12%coding
Personal finance8%finance
Scheduling and calendar6%scheduling
Cooking and meal planning6%cooking
Research and fact synthesis8%research
News and current events4%news
Travel planning6%travel
Writing and communication8%writing
Tutoring and education6%education
Health information4%health
Home and household ops4%home
Smart home / IoT4%smart_home
Shopping and gifts4%shopping
Music and media curation4%media
Photo and file curation3%photo
Interpersonal advice3%interpersonal
Legal and contract review3%legal
Vehicle / repair diagnosis2%vehicle
Civic / non-profit / charity3%civic
Customer support2%support

We use a crafted prompt and a parallel generation pipeline to generate the dataset in the AgenticML format. We then use a follow-on script to convert the trajectories into ChatML, which is used to train the base ChatML model for comparison. Once completed, I pushed the training data to the Hugging Face Hub, where it can serve as a foundation for future work. The dataset can be found here.

Run the training pipeline with the new agent template

For training, here is how we set up our experiment configuration and objectives:

Experimental Setup

ItemValue
BaseLlama 3.1 8B
Datakosiasuzu/agenticml-agent-trajectory-dataset (frames / messages paired)
InitReserved-slot embeddings (AgenticML) vs instruct tokenizer rows (ChatML)
TrainingLoRA r=32, α=64, 2 epochs, cosine LR 2e-4, batch 1 × grad accum 32
Modelsagenticml-llama3.1-8b-lora-merged, chatml-llama3.1-8b-lora-merged
PodRunPod w7ldz9cztwcljf, NVIDIA L40S
Commandagenticml eval-run-all --suites bfcl toolbench format_validity --continue-on-error
BFCL subset45 pinned IDs in src/agenticml/evaluation/benchmarks/bfcl/subset.py, seed 42
ToolBench subset10 pinned G1_instruction query IDs in toolbench/subset.py

Research Questions

  1. RQ1 — Format Validity: Can each format produce parseable, structurally valid agent output?
  2. RQ2 — Downstream benchmarks: Does serialization affect BFCL / ToolBench under matched SFT?
  3. RQ3 — Efficiency: Token and latency cost per successful BFCL task?

Here is a code snippet of the training configurations we used for the LoRA training of both models.



def build_lora_model(model_id: str):
    model = AutoModelForCausalLM.from_pretrained(
        model_id,
        dtype=torch.bfloat16,
    )
    model.gradient_checkpointing_enable()
    model.enable_input_require_grads()

    lora_cfg = LoraConfig(
        r=32,
        lora_alpha=64,
        lora_dropout=0.05,
        bias="none",
        task_type="CAUSAL_LM",
        target_modules=[
            "q_proj",
            "k_proj",
            "v_proj",
            "o_proj",
            "gate_proj",
            "up_proj",
            "down_proj",
            "lm_head",
        ],
    )
    return get_peft_model(model, lora_cfg)


def build_full_model(model_id: str):
    model = AutoModelForCausalLM.from_pretrained(
        model_id,
        dtype=torch.bfloat16,
    )
    model.gradient_checkpointing_enable()
    return model


def build_model(model_id: str, mode: TrainingMode):
    if mode is TrainingMode.LORA:
        return build_lora_model(model_id)
    return build_full_model(model_id)


def print_trainable(model) -> None:
    trainable = 0
    total = 0
    for p in model.parameters():
        total += p.numel()
        if p.requires_grad:
            trainable += p.numel()
    pct = 100 * trainable / total
    print(f"trainable params: {trainable:,} / {total:,} ({pct:.2f}%)")

We trained both the AgenticML model and the ChatML model on the full datasets, and each took about 2.5 hours on an L40S GPU.

AgenticML Training

AgenticML & ChatML Training

AgenticML Evals

AgenticML & ChatML Evaluations

Once the models were trained, I pushed both to the Hugging Face Hub: the AgenticML model and the ChatML model. We also deployed the LoRA adapters here and here.

Evaluating our format on BFCL and ToolBench against existing formats like ChatML

We built evaluation sets on top of ToolBench, Berkeley Function Calling Leaderboard (BFCL) and SWEBench-Mini. These benchmarks were chosen because they are representative benchmarks of agentic tasks. I had to drop SWEBench-Mini due to the large computational requirements needed to run it, so I focused on ToolBench and BFCL. I had to write custom bridges for the evaluations and scoring algorithms to map the ChatML format used by the evaluation benchmarks to the AgenticML format. We also created a format-validity benchmark to ensure the trained models were outputting responses in the correct format. Here are the results of the experiments.

SuiteFormatModelnPrimarySecondaryAvg TokensTok/SuccessAvg Wall Sec
BFCLAgenticMLagenticml-llama3.1-8b-lora-merged454.4% (accuracy)3.9 (avg_retry_count)140,0181,363223.9
BFCLChatMLchatml-llama3.1-8b-lora-merged4560.0% (accuracy)1.6 (avg_retry_count)14,2683,7753.9
ToolBenchAgenticMLagenticml-llama3.1-8b-lora-merged100.0% (pass_rate)1.0 (avg_steps)2,121—29.7
ToolBenchChatMLchatml-llama3.1-8b-lora-merged100.0% (pass_rate)9.8 (avg_steps)21,542—11.4
Format ValidityAgenticMLagenticml-llama3.1-8b-lora-mergedsmoke100% (valid_rate)100% (parse_rate)———
Format ValidityChatMLchatml-llama3.1-8b-lora-mergedsmoke100% (valid_rate)100% (parse_rate)———
Suite Primary Metrics

Suite Primary Metrics

Here is the breakdown of BFCL by category:

CategoryAgenticMLChatML
simple_python0/1212/12
multiple0/44/4
parallel0/40/4
parallel_multiple2/40/4
live_simple0/74/7
live_multiple0/66/6
multi_turn_base0/81/8

Here is a breakdown of the failures in BFCL:

43 failed tasks:

ShapeCountMeaning
json_tool_call28Model emitted tool JSON; BFCL grader still rejected
multi_turn_all_empty8No usable turn output across multi-turn loop
empty_no_tool_call7Encoded result was []

This is a breakdown of the failure modes for AgenticML in the evaluation:

  1. Duplicate parallel arrays on single-call tasks — e.g. simple_python_52, _57, _71, _114. Ground truth expects one {"name": ..., "parameters": ...}. Multiple action frames in one step → actions_to_result builds a JSON array. ChatML emits a single FC object → 12/12 on simple_python.
  2. Multi-turn collapse (0/8) — Loop stops when a step returns [] or parse error.
  3. Live schema mismatch (0/13 AgenticML vs 10/13 ChatML) — Live categories need exact dotted tool names; ChatML uses BFCL’s native system prompt; AgenticML uses goal/mission + TypeScript with_tool_obs.
  4. Parallel categories — ChatML often emits one call when several are required; AgenticML sometimes over-calls. parallel_multiple 2/4 is the one category where AgenticML beat ChatML.
BFCL Failure per Category

BFCL Failure per Category

All 10 ToolBench tasks completed inference and scoring. Every row failed structurally with no finish (no terminal finish action within step budget). Infra is fine; models need better multi-step tool use on G1_instruction.

AgenticML BFCL runs averaged 140k tokens and 224s wall time per task vs ChatML 14k tokens and 3.9s — largely driven by retry loops and multi-turn empties, not raw inference speed alone.

BFCL Efficiency Comparison

BFCL Efficiency Comparison

Conclusion

While I did not get the results I was expecting, I think this serves as a foundation for a new way of thinking about how we build agentic models. With more compute resources, I would be able to pursue some of these gaps in the performance of the AgenticML format, explore how to build custom evaluations for AgenticML and convert existing evaluations in ChatML into the AgenticML format for fairer evaluations. Here are some brief AI-generated takeaways from the work so far:

Prompt alignment vs BFCL: ChatML eval uses BFCL’s native ChatML + function-call path. AgenticML uses our frame runtime and bridge into gorilla result JSON. That asymmetry is intentional (each format evaluated in its natural harness) but it explains much of the live-category gap.

Eval fairness: Treating AgenticML 4.4% as pure capability understates the format. A harness fix (single-call collapse for non-parallel categories) is the highest-leverage ablation before retraining.

When AgenticML may still help: Trajectory clarity, custom runtimes, SWE-style long horizons, domains where ChatML FC is not the target wire format.

Limitations: 8B only; synthetic data; Format Validity not at n=100 on RunPod; SWE-bench skipped; no harness-fix ablation yet; ToolBench 0% both formats.

The entire code repository for the work can be found here: https://github.com/asuzukosi/agenticml

Last updated at: June 7, 2026