Optimizing Ruby Path Methods

To reduce CI setup time for the Intercom monolith, the author digs into Ruby boot performance, analyzing just how expensive require path resolution and directory scanning can be.

When Ruby does require, it linearly scans $LOAD_PATH and repeatedly checks for file existence, so as gems increase, boot cost grows in an O(N*M) shape. Bootsnap mitigates this with a load path cache. It scans $LOAD_PATH once to build a map of requirable files, then afterward uses hash lookups to turn relative paths into absolute paths, bypassing Ruby's slow search.

However, the cache is hard to invalidate. Bootsnap records directory mtime to revalidate, but since changes to child files don't propagate to the parent directory as a whole, multiple directories need to be recursively re-checked. In CI, git checkouts often don't preserve mtime, so rescan performance matters more than cache reuse.

The core bottleneck was an N+1 syscall pattern of calling File.directory? on every item during directory traversal. This arises because Ruby doesn't expose the d_type information that readdir(3) on Linux/BSD provides to the Dir.foreach block. To fix this, the author proposed a Dir.scan API, and also built a prototype alternative where Dir.foreach passes type information as a second argument.

The initial prototype made recursive directory traversal roughly 2x faster. Afterward, nobu proposed a more natural design that passes a File::Stat object, and it was eventually settled that Ruby would go with a new method in the form of Dir.scan.

In the same area, File.join was also revisited, noting that the most common case is effectively just string concatenation. In benchmarks comparing it against a simple interpolation-based implementation, File.join was slower, and this too showed room for improvement visible in the boot profile. As a result, Bootsnap now scans about 32k files across 10k repositories in 230ms, a big improvement over the previous implementation's 500ms.