Local NVMe Storage vs Network Storage: An Honest Comparison

Local NVMe is an SSD attached directly over PCIe inside the same host your VM runs on. Network storage — whether SAN, distributed Ceph or a cloud volume — lives on separate systems reached over a network. This single difference in the data path explains almost every pro and con that follows. It also explains why the two can look identical on a spec sheet and feel completely different in production.

Local NVMe storage: what it is, and what it is not

Local NVMe storage is flash that sits physically in the same machine as the CPU running your workload, addressed over PCIe using the NVMe protocol. No read or write leaves the chassis: the kernel places the command in a submission queue in host memory, rings a doorbell register, the controller fetches the command over PCIe and posts the result to a completion queue. There is no network stack, no switch, no third-party storage software and no second computer that also has to answer.

The confusion comes from marketing, where "NVMe" often describes the media inside a storage cluster rather than the topology. A volume can be built entirely from NVMe SSDs and still sit three network hops away. What sets your latency is not the form factor of the cells, but the length of the path to them.

• It is local when the drive is in the same host and the only thing in between is the hypervisor — nothing else.
• It is not local when the wording is "NVMe-backed", "NVMe-based block storage", "high-performance volume" or "distributed NVMe cluster". Those describe media, not path.
• Also not local: anything you can detach and reattach to a different VM while it is running. That decoupling is exactly what defines network storage — and exactly what local NVMe, by definition, does not have.
• Grey area: NVMe over Fabrics with RDMA. Technically network storage, but far closer to local NVMe than classic iSCSI. The gap is a property of the path and its contention, not a law of physics.

How to check whether your storage is really local

From inside a guest this is harder to answer than it sounds, but a handful of commands narrow it down a long way:

1. lsblk -d -o NAME,MODEL,TRAN,ROTA,SIZE — shows transport and model. A genuine NVMe device appears as nvme0n1 and reports TRAN=nvme.
2. nvme list (package nvme-cli) — lists NVMe namespaces with their model strings. It only returns something if the namespace is actually exposed to you.
3. ls -l /dev/disk/by-path/ — a pci- path suggests a real PCIe device (bare metal or passthrough), a virtio-pci path suggests a hypervisor abstraction.
4. lsblk | grep -E 'rbd|nbd|dm-' and iscsiadm -m session — surface Ceph RBD, NBD and iSCSI attachments, all of which unambiguously cross a network.
5. Only meaningful on bare metal: ip -s link before and after a heavy write benchmark. If the NIC counters scale with your write throughput, the storage is on the network.

Now the honest caveat: in a typical KVM guest using virtio-blk you see a device called vda and cannot determine from inside what is behind it. Storage traffic leaves the host over the host's own NIC, invisible to you. That leaves exactly two reliable options: ask the provider, and measure. Measuring is the better of the two, because it does not tell you the topology — it tells you the thing you actually care about, which is how the path behaves under your load.

The data path: PCIe versus a network hop

With local NVMe the path to the data is short: CPU, RAM and flash sit in the same machine, the command travels over PCIe and the answer comes back the same way. With network storage, every single IO adds a chain — the request is serialised, crosses the network stack, the NIC and at least one switch, arrives at a storage node with its own IO scheduler and its own drives, and the acknowledgement has to travel the whole chain back.

On replicated systems there is another round on top: a write goes to the primary node, which forwards it to its replicas, and is only acknowledged once the replicas have acknowledged. Every one of those stops has its own queue, and time spent in a queue is not constant — it moves with utilisation. That is precisely where the outliers come from that an average never shows you.

Latency, throughput and queue depth

For lots of small, random accesses, latency matters most — and local NVMe has orders-of-magnitude lower latency because there are no network hops in between. On sequential throughput, well-connected network storage can come much closer, but it stays capped by the uplink.

The most important and least understood variable is queue depth. At high queue depth many requests are in flight at once, the round trips overlap, and the network path can largely be hidden. That is exactly why network storage often looks excellent in throughput benchmarks. At queue depth 1, hiding is impossible: exactly one request is outstanding and you wait the full round trip. A database commit is a queue-depth-1 event. Anyone who only benchmarks deep queues never measures what their users feel.

fsync, durability and the database commit path

This is where it gets concrete. A database may not acknowledge a COMMIT until its write-ahead log is durable on the medium. PostgreSQL writes the WAL and then calls fdatasync; MySQL/InnoDB does the same with innodb_flush_log_at_trx_commit=1. That call is not a formality — it forces a cache flush down to stable media and only returns once the lowest layer has confirmed.

On local NVMe that lowest layer is a flush command over PCIe to the controller. On network storage the flush has to cross the network to the storage node, become durable there, possibly become durable on every replica, and the acknowledgement has to come all the way back. And because each of a single client's transactions is serialised behind that call, an uncomfortable rule applies: one client's transaction rate is bounded by its commit latency. No faster CPU and no extra RAM change that.

This is why teams eventually reach for synchronous_commit=off or innodb_flush_log_at_trx_commit=2. The application does get faster immediately — by trading durability for latency: on a crash, recently acknowledged transactions are gone. That can be a deliberate and correct decision. It is rarely a good one when it is made unconsciously because the storage was too slow.

Tail latency: why p99 matters more than the average

The mean of an IO measurement is set by the large mass of unremarkable requests. Your users do not experience the mean, they experience the outliers. And outliers are made where a queue is full: the closer a shared system gets to saturation, the more disproportionately waiting time inside it grows — and on a shared storage cluster, that utilisation includes the share taken by neighbours you cannot see and cannot control.

There is an amplification effect on top. If a single page view triggers twenty storage operations, it waits for the slowest of them. The device's p99 quietly becomes the page's typical experience. That is why the distance between p50 and p99.9 is the most informative number in a run — not the average.

Measure it yourself: the fio commands

Everything above can be checked on any host in half an hour. Install fio (apt install fio), make sure you have free space, and put the test file somewhere that holds nothing important. Important: never point --filename at a raw device that has data on it — the write tests overwrite whatever you aim them at.

1. Lay out the test file once

1. fio --name=layout --filename=/var/lib/fio.test --size=8G --rw=write --bs=1M --ioengine=libaio --direct=1 --iodepth=16

The file has to be larger than every cache in the path — the guest page cache, the host page cache and the storage controller's cache. Otherwise you are measuring RAM, not storage.

2. 4k random read at queue depth 1 — the latency question

1. fio --name=randread-qd1 --filename=/var/lib/fio.test --size=8G --rw=randread --bs=4k --ioengine=libaio --direct=1 --iodepth=1 --numjobs=1 --time_based --runtime=300 --ramp_time=30 --percentile_list=50:90:99:99.9:99.99 --status-interval=10 --group_reporting

3. The same at deep queue — the throughput question

1. fio --name=randread-qd32 --filename=/var/lib/fio.test --size=8G --rw=randread --bs=4k --ioengine=libaio --direct=1 --iodepth=32 --numjobs=4 --time_based --runtime=300 --ramp_time=30 --percentile_list=50:90:99:99.9 --status-interval=10 --group_reporting

4. 4k random write at queue depth 1

1. fio --name=randwrite-qd1 --filename=/var/lib/fio.test --size=8G --rw=randwrite --bs=4k --ioengine=libaio --direct=1 --iodepth=1 --numjobs=1 --time_based --runtime=300 --ramp_time=30 --percentile_list=50:90:99:99.9:99.99 --status-interval=10 --group_reporting

5. The fsync-bound test — the commit path

1. fio --name=commit --filename=/var/lib/fio.test --size=1G --rw=randwrite --bs=4k --ioengine=psync --fdatasync=1 --iodepth=1 --numjobs=1 --time_based --runtime=300 --ramp_time=30 --percentile_list=50:90:99:99.9 --group_reporting
2. Repeat with --numjobs=16 to model sixteen connections committing concurrently.
3. As a PostgreSQL-specific cross-check: pg_test_fsync — it times the same operation for every available wal_sync_method.

--direct=1 is deliberately absent here. A database writes its WAL buffered and then forces durability with fdatasync, and --ioengine=psync --fdatasync=1 reproduces exactly that sequence. fio reports the time spent in those sync calls as its own block — that block is the one you care about, not the write latency printed above it.

How to read the output

• clat is completion latency, slat is submission latency, lat is the sum. For comparing storage, read clat — slat mostly tells you about your own kernel path.
• Read the clat percentiles at queue depth 1. That is the time a synchronous commit genuinely experiences. The IOPS figure at queue depth 32 answers a different question entirely.
• Compare p50 against p99.9, not against the mean. A tight spread means a predictable path; a wide spread means you are contending for a queue somewhere along it.
• A high standard deviation is the same signal in compact form: the path is not exclusively yours.
• numjobs and iodepth model different things. numjobs are threads, so clients; iodepth is outstanding IOs per thread, so batching. Databases look like many jobs at shallow depth, not one job at depth 64.
• A single 30-second run hides noisy neighbours completely. Contention arrives in bursts: it moves p99.9 and the interim lines from --status-interval, and barely moves the mean. Run long, repeat at different times of day, and compare runs against each other.
• Many platforms use burst credits. A short test runs entirely inside the credit budget and therefore measures the best case rather than the typical one. --ramp_time helps you skip the warm-up; only a long runtime exhausts a burst budget.
• Do not measure on an idle box. Run the tests while your real workload is running — what matters is how the storage behaves in your actual conditions, not in a lab.
• And the most useful rule for reading someone else's spec sheet: an IOPS number without block size, queue depth, read/write mix and whether --direct=1 was set is not a claim about anything.

Where network storage genuinely wins

Here network storage wins fair and square — and not on a single point either. If you actually use the capabilities below, the extra latency buys you something concrete rather than being a mere compromise:

• Decoupled persistence: when the compute host dies, you attach the volume to another machine. With local NVMe you restore from backup instead — a completely different recovery time.
• Live migration: hardware maintenance without downtime for you. With local NVMe, host maintenance means a reboot window or an actual data move.
• Copy-on-write snapshots and clones: twenty identical test environments from one golden volume cost almost nothing. Locally you copy bytes for that.
• Capacity beyond the host: a volume can grow without touching the compute size, and beyond what physically fits in one server.
• Built-in redundancy: replication or erasure coding deliver durability without a line of application code. With local NVMe that is your job — RAID protects against a drive, not against a host.
• Operational simplicity at fleet scale: stateless compute is easier to automate, autoscale and simply rebuild.
• And the most honest point: for many workloads the latency is simply irrelevant. A shop serving from cache, a low-write CMS, archives, mail spools, asset stores — paying a latency tax nobody notices in exchange for high availability you genuinely use is the right call.

Noisy neighbours — and why "local" alone guarantees nothing

A shared storage cluster has the same underlying problem as an overbooked host: when one neighbour floods the storage with IO, everyone else feels it. This is the storage flavour of overprovisioning — just for IOPS instead of CPU cycles. Local NVMe you share, at most, with the few VMs on the same host, not with an entire cluster.

Even so: "local" is not a magic word. A host that stacks forty VMs onto one drive produces outliers just as ugly as a saturated SAN — the bottleneck simply moves from the network into the device. The word "local" on a spec sheet guarantees topology, not performance. What matters is the combination of a short path and a commitment that the host is not overprovisioned. That is why we state those two things together rather than separately.

Why local NVMe wins for databases and builds

• Databases (PostgreSQL, MySQL): every commit is an fsync at queue depth 1 — precisely the discipline in which a network hop cannot be hidden. Commit latency drops, and with it the ceiling on transactions per connection.
• CI/CD and build servers: tens of thousands of small file operations for node_modules, compiler caches or container layers are almost pure metadata load, and outliers accumulate across the whole pipeline.
• Caches and search indexes (Redis persistence, Elasticsearch): an AOF rewrite or a segment merge is a write storm, and on shared storage it lands exactly when response times need to stay calm.
• Analytics workloads: high sequential throughput without a network bottleneck and without competing for the same uplink.
• Anything stateful whose p99 is visible to a user: message queues, session stores, event logs.

How to choose

• Choose local NVMe when your workload writes synchronously and the response time is visible — and plan backups from day one, not later.
• Choose network storage when recovery time, live migration, snapshots or capacity beyond a single host are your real requirements.
• Choose both where you can: WAL and hot data local, cold data, archives and backups on decoupled storage. That is the split most mature setups converge on anyway.
• And do not decide from a spec sheet: run the fio commands above on both candidates, with your block size and your access pattern.