Two Planes of Truth: Execution Fidelity and Behavioral Correctness in Tool-Using LLM Agents

Abstract

The increasing integration of large language models (LLMs) with external tools has created a need for evaluation approaches that extend beyond final-answer accuracy and examine whether agents execute tasks through valid and dependable operational processes. This study proposes a two-plane evaluation framework that distinguishes execution fidelity, representing the correctness of tool selection, argument construction, execution order, constraint adherence, observation utilization, and error recovery, from behavioral correctness, representing the accuracy, completeness, validity, and instruction conformity of the resulting output. A controlled benchmark comprising 2,160 agent–task executions was constructed across four LLM agents, six categories of tool-mediated tasks, and three levels of task complexity. Overall, 33.6% of executions were faithful-correct, whereas 15.6% were unfaithful-correct, demonstrating that behaviorally successful outcomes may conceal deficiencies in the underlying execution trajectory. A further 4.4% were faithful-incorrect and 46.5% were unfaithful-incorrect. Execution Fidelity Score (EFS) and Behavioral Correctness Score (BCS) were strongly associated (Spearman’s ρ = 0.835, p < 0.001), yet their divergence across individual runs confirmed that they represent related but non-equivalent dimensions of agent performance. Task complexity strongly affected both EFS and BCS, with faithful-correct performance declining from 65.0% under low-complexity tasks to 5.42% under high-complexity tasks. Trace-level analysis further revealed task-specific failure patterns, particularly in sequencing, argument construction, output formatting, observation interpretation, and recovery. The findings demonstrate that outcome-only evaluation can overestimate the reliability of tool-using agents by overlooking procedurally deficient successes. The proposed two-plane framework provides a more transparent and diagnostically informative basis for evaluating trustworthy, auditable, and robust LLM agents in increasingly complex tool-mediated environments.