Do You Really Need a Database

Data ultimately sits on top of a file. The core question isn't whether you write a file, but whether you use a hand-built file format or a DB's file structure.

Three storage approaches are compared.

• Scanning the entire file on every request: Open a JSONL file like users.jsonl and parse it line by line to find the ID. The structure is the simplest, but at O(n) it slows down sharply as data grows.
• Loading into memory at startup: Read the file once into a map/HashMap, and writes update both memory and the file simultaneously. Reads are O(1), and concurrent reads hold up well with RWMutex/RwLock.
• Binary search on disk: Keep a data file sorted by ID and a fixed-width index file, then perform binary search using ReadAt. You don't need to load everything into memory, and lookups are O(log n).

For the benchmark, the same HTTP server was built in Go, Bun, and Rust, and measured with wrk for 10 seconds. The datasets were 10k / 100k / 1M records. Additionally, SQLite (modernc.org/sqlite) was also compared in Go.

The results are clear.

• Linear scan collapses as data grows. At 1M records, Go drops to around 23 rps and Bun to around 19 rps.
• In-memory map is the fastest. Go reaches about 97k rps, Bun 106k rps, and Rust up to 169k rps. Latency is also sub-millisecond.
• Disk-based binary search is stronger than expected. In Go, it delivers roughly 39k–46k rps with average latency of 1.2–1.4ms, staying nearly flat even as scale increases.
• SQLite is also stable, at around 25k–26k rps with average latency of about 2ms, and it's less sensitive to data size.
• In this test, a hand-written sorted file + index is about 1.7x faster than SQLite. Looking purely at simple key lookups, you can see the cost of a DB engine's generality.

The practical conclusions are as follows.

• If you need the highest throughput and can fit it in RAM: in-memory map
• If you need fast lookups without RAM: sorted file + disk binary search
• If there's a possibility you'll need SQL later: SQLite
• If your goal is building it fastest: you can start with linear scan, but it hits limits as soon as scale grows even a little

Even though 25,000 rps looks like a big number, when you work backward from daily traffic, peak ratios, and lookups per user, even smaller-than-expected services can fall into this range. In the end, choosing a storage method is less about "whether you have a DB or not" and more about matching your current scale and future query patterns.