Background & NFS Security Challenges
Imagine a cluster of five web servers that all need read and write access to a 500GB media repository. Storing copies locally on each server wastes disk space. Continuously syncing with rsync causes network congestion and data drift.
NFS solves this problem cleanly. It allows you to mount a remote directory directly to your local system. Applications read and write files over the network seamlessly, perceiving no difference compared to an internal drive.
However, legacy NFS carries a major security risk. Under the default sec=sys mode, the server blindly trusts the IP address and UID provided by the client. An attacker needs only to plug an unauthorized laptop into the internal switch, spoof an allowed IP, and create a root user (UID 0) to wipe out the entire shared dataset.
To ensure security, you must use NFSv4 with Kerberos authentication (the sec=krb5p option). This approach delivers two critical layers of defense: requiring a valid identity ticket before mounting, and encrypting all in-transit traffic using AES to prevent packet sniffing.
Lab Environment & Installation
The lab setup consists of two Linux machines on the same local subnet:
- NFS Server:
192.168.1.10(hostname:nfs-server.lab.local) - NFS Client:
192.168.1.20(hostname:nfs-client.lab.local) - KDC Domain:
LAB.LOCAL
Critical Technical Note: Kerberos will reject any authentication ticket if the clock skew between systems exceeds 300 seconds (5 minutes). Make sure chrony or systemd-timesyncd is synchronizing time accurately. In addition, forward and reverse DNS (PTR records) must resolve properly for both server and client.
1. Installation on NFS Server
Update repositories and install the required packages for Ubuntu/Debian:
sudo apt update
sudo apt install -y nfs-kernel-server krb5-userFor environments running RHEL or AlmaLinux 9:
sudo dnf install -y nfs-utils krb5-workstation2. Installation on Client
Perform a similar installation on the client machine:
# Ubuntu / Debian
sudo apt update
sudo apt install -y nfs-common krb5-user
# RHEL / AlmaLinux
sudo dnf install -y nfs-utils krb5-workstationDetailed Configuration
1. Create Kerberos Principals and Export Keytabs
Each node requires a service principal in the format nfs/<FQDN>@REALM. This is stored in /etc/krb5.keytab so the server and client can mutually authenticate without entering passwords.
Create the principals directly on the KDC server:
# Create principals for both machines
kadmin.local -q "addprinc -randkey nfs/[email protected]"
kadmin.local -q "addprinc -randkey nfs/[email protected]"
# Export keytabs separately
kadmin.local -q "ktadd -k /tmp/server.keytab nfs/[email protected]"
kadmin.local -q "ktadd -k /tmp/client.keytab nfs/[email protected]"Transfer each keytab file to /etc/krb5.keytab on its respective machine over secure SCP, then restrict file permissions:
sudo chmod 600 /etc/krb5.keytab
sudo chown root:root /etc/krb5.keytabSynchronize the NFSv4 domain mapping in /etc/idmapd.conf on both machines:
[General]
Domain = lab.local2. Configure the Shared Directory on Server
Create the actual storage directory and set read/write permissions:
sudo mkdir -p /srv/nfs_share/project_data
sudo chmod -R 775 /srv/nfs_share/project_dataDeclare the exported directory in /etc/exports:
sudo nano /etc/exportsAdd the following configuration line:
/srv/nfs_share/project_data 192.168.1.0/24(rw,sync,no_subtree_check,sec=krb5p)Explanation of the configuration options:
rw: Grants read and write permissions to the client.sync: Replies to requests only after changes have been committed to physical disk, reducing the risk of corruption during unexpected power outages.no_subtree_check: Prevents subtree checking against parent directory hierarchies, significantly speeding up file access.sec=krb5p: The highest security level (Privacy). Both data payloads and packet headers are encrypted end-to-end with AES across the wire.
Reload the export table and start the rpc-gssd service to bridge Kerberos:
sudo exportfs -rav
sudo systemctl restart rpc-gssd
sudo systemctl enable --now nfs-kernel-server3. Mount the Share on the Client
The client must run the rpc-gssd daemon to handle Kerberos tickets during mounts:
sudo systemctl enable --now rpc-gssdCreate the mount point and perform a manual mount to verify:
sudo mkdir -p /mnt/shared_data
sudo mount -t nfs4 -o sec=krb5p nfs-server.lab.local:/srv/nfs_share/project_data /mnt/shared_dataConfigure persistent automounting on boot by adding an entry to /etc/fstab:
nfs-server.lab.local:/srv/nfs_share/project_data /mnt/shared_data nfs4 sec=krb5p,_netdev,rsize=1048576,wsize=1048576 0 0The _netdev option delays mounting until network interfaces are fully up with assigned IPs. Without this flag, the Linux boot process can hang.
The rsize=1048576,wsize=1048576 parameters (1MB buffer) make a dramatic performance difference. In testing with a 10GB tar file over a 1Gbps network, default buffer sizes yielded only 35–40 MB/s. Increasing the buffer to 1MB drove throughput to 112 MB/s, fully saturating the network link.
Testing & Performance Monitoring
1. Verify Write Access and Kerberos Ticket Handling
Check the status of the mounted share:
df -hT /mnt/shared_dataObtain a TGT ticket for a user account and write a test file:
# Request a Kerberos ticket for the user
kinit [email protected]
# Write a test file to the network share
echo "NFS Kerberos testing OK" > /mnt/shared_data/test_sync.txt
cat /mnt/shared_data/test_sync.txtDestroy the ticket using kdestroy and attempt to modify the file again. The system will immediately return a Permission denied error. This confirms that access is strictly gated by an active Kerberos session rather than relying purely on IP addresses.
2. Monitor I/O and Latency with nfsstat and nfsiostat
Inspect total RPC requests and cache efficiency:
nfsstat -s # View server-side statistics
nfsstat -c # View client-side statisticsInstall the sysstat package on the client to use nfsiostat:
nfsiostat 2 5This command gathers metrics every 2 seconds across 5 reporting intervals. Pay special attention to these three metrics:
- ops/s: Number of read/write operations per second.
- kB/s: Actual data throughput pushed over the network interface.
- avg RTT (Round Trip Time): Duration from when the client sends an RPC request until the server completes its response.
If avg RTT exceeds 20ms on a local network, either the storage backend is suffering from disk I/O bottlenecks or the network switch is dropping packets. Check the I/O wait metrics on the server with iostat -xz 1 to isolate the bottleneck.
