Your Network Is Not a Private Superhighway
The most common NFS tuning guides are written from the perspective of a clean, isolated lab. In this perfect world, you have two machines connected by a high-speed switch with nothing else to do. Production is the opposite. It’s a chaotic public road
system at rush hour. Your NFS traffic is competing with database queries, user web traffic, backup jobs, and countless other services. This congestion introduces latency and the possibility of dropped packets. A large read/write size (like the often-recommended 1MB `rsize`/`wsize`) that flies on a clean network can cause packet fragmentation and retransmissions on a congested one, killing performance. Tools like `nfsstat` can reveal if you're suffering from a high number of retransmissions, suggesting that your theoretically optimal settings are being undermined by real-world network conditions. The pristine lab has near-zero latency; your production network, with its layers of switches, routers, and firewalls, does not.
Production Workloads Are Messy and Unpredictable
Lab benchmarks are simple. You run a command to write a single, large file to test sequential throughput, or you read thousands of tiny files to test IOPS. It's predictable. A production application workload is a chaotic mix of everything all at once. A web application might be serving large images (big reads), while users upload small avatars (small writes), and the application itself logs constant status updates (tiny, metadata-heavy writes). Generic tuning advice often focuses on optimizing for one type of workload, usually large file transfers. But in production, an application that performs lots of file locking or frequently checks file attributes can be brought to its knees, as these operations force the system into a slower, more careful synchronous mode to ensure data consistency. Optimizing for one type of I/O can inadvertently penalize another, making the overall application slower.
The 'sync' vs. 'async' Data Integrity Dilemma
For benchmark glory, setting your NFS server exports to `async` (asynchronous) is a classic trick. It allows the server to tell the client, “Got it!” as soon as it receives a write request in memory, before the data is safely on disk. This makes writes feel instantaneous. The problem? If that server loses power or crashes before the data is physically written, that data is gone forever. In a production environment with critical data, this is often an unacceptable risk. Most production systems default to `sync` (synchronous) writes, where the server doesn't confirm the write until the data is on stable storage. This is much slower but guarantees data integrity. While some guides praise `async` for its speed, they often fail to mention the high-stakes gamble you're taking in a production context. Unless your application is designed to handle potential data loss, blindly enabling `async` is a recipe for disaster.
Hidden Overhead from Storage and Virtualization
In a lab, your NFS server is likely a dedicated machine with its own disks. In production, that server might be a virtual machine sharing CPU and memory with a dozen other VMs. Your storage might not be a simple set of disks but a complex Storage Area Network (SAN) or cloud-based file system serving hundreds of clients. This introduces layers of “noisy neighbors” and resource contention. The default number of NFS server threads on many Linux systems is shockingly low, sometimes as few as four, which is easily overwhelmed in a busy environment. Increasing this number is a common production-only fix. Furthermore, features like storage-level compression or deduplication can add latency, and a nearly full file system can dramatically slow down write operations as the system struggles to find free space. None of these factors are present in a simple test, but they all conspire to slow things down in the real world.
