NFS server behind a PF firewall
Second post entitled something behind some-other-thing, I've wasted all my imagination when I was a child... ;)
Yesterday I found a small problem with NFS while working on this setup:
In the setup, there are 2 Linux-based NAS devices that export a 1Tb share over NFS. Both NAS are in sync so, if one fails, we can mount the other one and users would be able to keep working while the first NAS is replaced.
Both NAS devices are in a Gigabit LAN, connected with one of the network interfaces on a FreeBSD server (subnet 192.168.10.0/24).
On the other side of the server, there was another LAN, where workstations are connected to the server (subnet 192.168.1.0/24).
The plan was to mount the /nfs/shares share in the FreeBSD server and then export it again from the server, allowing the workstations to mount it. It didn't work. After some reading I found out that NFS does not like to re-export NFS shares, that is, if you mount an NFS share from server A on server B and then you try to export that share from the mount point in server B to server C (for example) you will get all kinds of nasty errors.
It was time for a change in the plan. I didn't want to give full-access to the NAS devices and I didn't want the workstations to mount the share directly from the NAS devices either, because if one fails, I would have to modify the mounts in every workstation so they use the backup NAS device.
Perhaps I could use PF in the FreeBSD server to redirect NFS traffic from the workstations directly to the NAS devices...
Well, it took me some time to think about how to do it (and some help from viq at #pf in freenode). The best approach could be some kind of binat that could map the NFS requests for a given ip address to the ip address of the selected NAS device. Anyway, you can't map a whole network (or any to use a wildcard), so binat would not work. I had to think about a different way to do it.
Let's begin modifying the setup a little bit:
As you can see in the picture, my idea was to add an alias to the nic connected to the 192.168.1.0/24 subnet, then I would be able to use PF to redirect all incoming traffic for that alias (192.168.1.100) to a given NAS device. As I've a local DNS server in the network I could add an A entry for nfs.mydomain.com that points to 192.168.1.100, this way I would be able to use that internal subdomain instead the ip address to access the NFS shares. Nice, the new plan was finished, let's start it! (I'll omit the DNS Zone modification, as it is out of the scope of this post)
First, I added the alias definition to /etc/rc.conf:
ifconfig_bge0_alias0="inet 192.168.1.100 netmask 255.255.255.0"
This will create the alias properly after a reboot. Then I created the alias manually:
ifconfig bge0 alias 192.168.1.100
Once I've created the alias, I added PF support in /etc/rc.conf:
pf_enable="YES" pflog_enable="YES" gateway_enable="YES"
This will load the needed stuff on boot (kernel modules, sysctl knobs, etc). This will work if you've the GENERIC kernel, if not, you will have to check if your custom kernel has pf support enabled within the kernel or as a KLD.
Then I created the /etc/pf.conf file. This is a stripped version of it (It shows only the needed stuff for the NFS redirection to work):
lan_if="bge0" # NIC connected to the Intranet LAN
nas_if="bge1" # NIC connected to the NAS-storage LAN
nfsalias="192.168.1.100" # Local alias to manage requests for NFS storage
mainnas="192.168.10.2" # Main storage NAS
backupnas="192.168.10.3" # Backup storage NAS
rdr on $lan_if proto { tcp, udp } from $lan_if:network to $nfsalias port 111:65535 -> $mainnas
pass in log quick on $lan_if proto { tcp, udp } from $lan_if:network to $mainnas keep state
pass in all
pass out all
What I do with this pf.conf configuration file is set a redirection so all tcp and udp traffic from the workstations LAN to the alias ip on any port between 111 and 65535 will get redirected to the main NAS device.
The reason for such redirect is that NFS uses random ports for connections (in a similar way to what FTP does) so it is easier to open a full-range of ports. The first port is 111 as it is the first port needed by NFS (sunrpc/rpcbind) and, this way, I block requests for the NAS device web interface (port 80) and SSH (port 22) and users from the workstations LAN will not be able to access the management interfaces of the NAS device.
You should have noticed that the following line is not needed at all:
pass in log quick on $lan_if proto { tcp, udp } from $lan_if:network to $mainnas keep state
It is not (because I do a pass all later) but it allows me to log only NFS traffic (useful for debugging).
Ok, once I had everything in place, I started PF:
/etc/rc.d/pf start
(This will load the kernel module if needed, set the proper sysctl options and will enable pf too)
If my plan was right (and it was ;D) I should be able to access the NFS shares of the main NAS device from any workstation, so I did some checks using showmount:
$ showmount -d nfs.mydomain.com Directories on nfs.mydomain.com: *,192.168.1.0/24 *,192.168.10.1 /mnt/soho_storage/samba/shares/shares
It worked!. that soho_storage path you see there is the full path of the shares share within the NAS device filesystem. The Iomega NAS stores all the data within a directory called samba/shares and then it creates symlinks of the data within the /nfs directory.
Ok, as showmount worked, it was time for the final test, to mount the share. In FreeBSD workstations I modified /etc/fstab, adding:
nfs.mydomain.com:/nfs/shares /mnt/nfsstuff nfs rw,tcp 0 0
In the Linux workstations I added a line like:
nfs.mydomain.com:/nfs/shares /mnt/nfsstuff nfs rw,tcp,rsize=32768,wsize=32768,hard,intr,timeo=14,bg 0 0
Of course /mnt/nfsstuff must exist. With the proper fstab lines in place, I just tried to mount the share:
# mount /mnt/nfsstuff
Which worked as expected:
$ df -h Filesystem Size Used Avail Capacity Mounted on [ stripped ] nfs.mydomain.com:/nfs/shares 929G 269G 660G 29% /mnt/nfsstuff
Conclusion:
It was really disgusting to find out that NFS does not support re-export of shares. This is the first time in many years I find something I don't like about NFS. Anyway, I found a great solution, perhaps even better than re-exporting it. With this setup, if the main NAS device fails, I only have to change one PF rule and all the requests for nfs.mydomain.com will go to the backup NAS. If needed, I could even split NFS traffic, sending some workstations to the main NAS and some other to the backup NAS or I could block NFS traffic from certain ip addresses, adding some security to the whole thing. IMHO this is a great solution.