Building a Rails test automation agent that writes the tests developers don't

In large Rails monolith codebases, building new features tends to take priority over writing tests, and as time passes, unverified code piles up and debugging costs grow. To address this, an autonomous agent was built that reads source files and generates and improves RSpec tests, then self-validates until it passes style rules and coverage targets.

The agent operates on the premise that, because of Ruby's dynamic typing, even test syntax validation ultimately has to rely on execution. In a Rails codebase, each file type—model, serializer, controller, mailer, helper—needs a different testing approach, so separate guidance was required for each type. There are also exceptional mappings, such as app/controllers/ mapping to spec/requests/, so having a structure that could quickly find the correspondence between source files and spec files was important.

In the RSpec ecosystem, where a lot of code is reused, shared assets like factories, fixtures, and schemas were key variables. The agent was designed to create a new factory if none existed, reuse one if it already did, and modify shared files—which affect multiple tests—carefully.

The implementation was built on top of Mistral's open-source coding assistant Vibe. The default system prompt was left as-is; instead, three things were strengthened. A repository-level AGENTS.md enforces a step-by-step execution plan, dedicated SKILLs were set up per file location, and RuboCop and SimpleCov were attached as custom tools.

AGENTS.md was designed to follow this order:

• Read the source file
• Read documentation if it exists
• Check whether an existing spec exists
• Select and read exactly one skill matching the file's location
• Find existing patterns, factories, and helpers
• Run Extract → Factory → Generate tests
• Verify with RuboCop
• Verify with SimpleCov

It also included rules such as not using vague matchers like be_present, be_truthy, be_between, or include(:key), and using eq(exact_value) instead. Also, to reduce the problem of the agent missing some public methods, it was forced to re-read the source file at the end and self-check whether "all public methods have been tested." This single file alone raised the quality score from 0.68 to 0.74.

For each file type, dedicated skills were created separately, such as a request spec skill for controllers. Since a single general-purpose skill struggled to handle models and controllers the same way, having separate guidance per file category made the results more consistent.

The RuboCop tool lints spec files, and the SimpleCov tool returns both the RSpec run results and coverage together. SimpleCov was especially important because it returns not just whether tests pass, but also which source lines were actually never executed, allowing the agent to self-correct. Initially, only about 1/3 of the generated tests passed on the first run, but iterative fixes resolved most of the failures.

Test quality was evaluated based on the Arrange-Act-Assert principle, accurate assertions, and a balance between happy paths and error paths. However, since numbers alone—zero RuboCop violations, 100% coverage, all tests passing—couldn't determine whether a test was actually good, a separate LLM-as-a-judge score was introduced. Criteria such as "are all error conditions tested" and "is this a good test" were scored on a 0 to 1 scale, with the grading criteria clearly written out to reduce variance across models.

However, relying on the LLM score alone still left another problem: a test could look good yet fail to run at all due to a syntax error. For example, a test missing one closing parenthesis in User.new( looks structurally excellent but actually fails. So ultimately, executability verification and coverage verification need to run together, and the agent was designed to process many files in parallel, filling test gaps inside CI/CD without human intervention.