Tracing Agent Harness Call Paths with NVIDIA NeMo Relay

NVIDIA released a tutorial and a case study of Hermes Agent utilizing 'NeMo Relay', a runtime for tracing the internal operations of agent harnesses. NeMo Relay observes execution at the session, turn, and call levels without rewriting existing stacks, is developed in Rust, and supports Python and Node.js bindings.

Key Features and Output Formats

NeMo Relay records execution logs through Scope, Lifecycle Event, and Mark, offering three output formats: ATOF (JSONL, for debugging), ATIF (JSON, for analysis), and OpenTelemetry+OpenInference (span hierarchy). Hermes Agent has Relay built-in, while LangChain and others support observation, security, and optimization, and Codex and Claude Code integrate via a CLI sidecar approach.

Hermes ToolPerf Case Study Results

The impact of harness changes was analyzed using Nous Research's Hermes ToolPerf benchmark (9 failure-type tasks based on 1.5 million operational log messages). Comparing the baseline commit and the modified commit yielded the following results:

• Qwen3 Coder 30B: Success rate rose from 70% to 81%, but average LLM calls increased from 3.8 to 4.9, and duration increased from 27 seconds to 42 seconds.
• Detailed Analysis: A significant portion of the success rate increase was related to XML output errors (7 occurrences) caused by broken OpenRouter chat templates; excluding these, the difference in success rate is negligible. Conversely, a regression in the err_case_search task caused a sharp increase in LLM calls.
• Claude Sonnet 4.5: No substantial changes were observed.

Precautions for Application

• There may be configuration format differences between the tutorial pinned to NeMo Relay 0.8.3 and the latest version (0.9.3).
• Since traces include prompts and tool results, reviewing the content before saving or sharing is essential.
• When using public API gateways, provider errors may be aggregated as failures, so the actual termination step must be verified in the trace.