Reverify Released: A Hallucination Verification Tool for AI Binary Analysis

A tool called Reverify has been released to address the hallucination problem where language models, while good at reading source code, fabricate offsets or structures during binary analysis. This project is a collection of deterministic pure Python tools that compare AI model hypotheses with actual file bytes, ensuring output reliability by placing judgment authority in the tool rather than the model.

Core Features and Verification Method

Reverify performs PE, ELF, and Mach-O file parsing, x86/x64/ARM/ARM64 disassembly, and CPU emulation. It evaluates claims made by the model (such as byte locations or pattern existence) as one of VERIFIED, REFUTED, or INCONCLUSIVE. Only verified results are reported, while refuted claims are recorded as KNOWN FALSE to prevent recurrence. Additionally, an information-weighting system is applied to prevent repetition of trivial facts and to score only meaningful reconstruction results.

Benchmark and Performance

In benchmark tests, when models claimed textbook function entry patterns, they exhibited a 100% hallucination rate across 19 Windows system DLLs and 19 aarch64 ELF files. However, Reverify refuted all of them, with 0 false VERIFIED judgments. After a single feedback round, the models derived correct answers matching actual bytes in 100% of the files.

Usage and Installation

Reverify is released under the MIT License and can be installed via pip install reverify. It operates as a CLI and an MCP (Model Context Protocol) server, allowing AI agents like Claude Code and Cursor to call the tool directly. It can be installed and used without Ghidra, and mature engines such as Capstone, Unicorn, LIEF, and angr can be swapped in as backends if needed.