PORTool: Reward Tree-Based Importance-Aware Policy Optimization for Multi-Tool Integrated Reasoning

PORTool uses a rewarded rollout tree to reduce the credit assignment problem that arises from outcome-only reward training. Because the structure shares a common prefix before branching, it allows direct comparison of different tool-use decisions within the same context.

Step-wise importance is estimated using two signals.

• A correctness-dominant signal that checks whether the descendant paths of a given step can reach the final correct answer
• An auxiliary term that reflects whether the tool call at that step actually succeeded

The policy is updated using the step-wise importance obtained this way, guiding more efficient tool calls and reasoning. By jointly reflecting local branch comparisons and overall path quality, it moves toward reducing unnecessary calls while raising the correct-answer rate.

In experiments, final answer accuracy improved while the number of tool-call steps decreased, and the ablation study also confirmed the robustness of the proposed step-wise importance estimation.