Do You Really Need a Database

A database is ultimately a structured collection of files on top of a file system, so early-stage applications can be fast enough by managing files directly.

The same HTTP server was implemented in Go, Bun (TypeScript), and Rust to compare the performance of GET /users/:id lookups, using three main storage approaches.

• Scanning the file on every request: since each request reads and parses the JSONL file to the end, this is O(n), and it got drastically slower as the data grew.
• In-memory map: the entire file is loaded at startup into an ID-based hash map, so lookups are handled in O(1). This produced the highest throughput.
• On-disk binary search: with a sorted data file and a fixed-width index, lookups are done via ReadAt in O(log n). Without loading everything into RAM, this was a middle-ground solution that was still much faster than linear scanning.

The benchmarks were run with 10k / 100k / 1M records, a 10-second wrk test, on an Apple M1 Mac mini. For Go, an additional comparison was made against SQLite (modernc.org/sqlite).

The key results are as follows.

• Linear scan dropped to 23 req/s for Go and 19 req/s for Bun at 1M records.
• On-disk binary search had a small decline, going from 45k → 38k req/s across the 10k–1M range.
• SQLite was stable at around 25k req/s with an average latency of 2ms.
• The in-memory map was the fastest, recording 97k–169k req/s with latency under 0.5ms.
• Rust's in-memory map delivered the best performance (169k req/s), and Bun was slightly faster than Go.

Based on this, the author argues that most early-stage products can get by with just a single server and file storage, without a separate database. For simple services, a single SQLite file is seen as capable of handling even very high traffic.

However, the following conditions are laid out as the point where a database becomes necessary.

• When the data no longer fits in RAM
• When lookups on fields other than ID are needed
• When joins are needed
• When concurrent writes from multiple processes are needed
• When ACID guarantees are needed, such as atomic writes across entities

As an appendix, the server code for Go, Bun, and Rust along with the seed/benchmark scripts are provided, alongside an introduction to the DB Pro product and links to reference articles on SQLite, Postgres, and distributed SQLite.