Important things to know about NFS 3
In NFS 3, there are 2 access barriers:
- The NFS server configures exports based on network addresses, decides whether the client is allowed root access permissions, and whether an export is read+write or read-only for each client listed.
- Basix POSIX permissions (user, group, other) - the NFS server will enforce these basic permissions, but trusts the client to not lie about what your user id / group id are in its default configuration (AUTH_SYS).
- The main limitation of NFS 3 is that it does not natively support ACLs - you are absolutely stuck with user/group/other permissions in most cases unless the NFS server does magic under the hood.
- 📖 8.7.1. NFS Security with AUTH_SYS and Export Controls
The NFS server uses several ports to make NFS 3 available:
- tcp/udp 111 (portmapper)
- tcp/udp 2049 (nfs)
- statd, mountd, lockd, rquotad are assigned to dynamic ports
Important things to know about NFS 4
- NFS 4 inherits the security barriers from NFS 3 and adds access control lists - (ACLs). ACLs are more complex than the simple model and allow you more flexibility than the simple [user, group, everyone else] model that basic Linux/POSIX permissions allow. With an ACL, you layer in the basic permissions along with a list of additional users or groups and permissions. Any command that displays the ACL for an object generally also shows a representation of the basic permissions so you don’t need to run 2 commands to get a complete picture.
- While there is such a thing as a POSIX ACL, NFS 4 ACLs follow the Windows NT permission model instead. Posix ACLs are only cumulative - you could only ever gain more permissions from them. There is no concept of a deny - except in cases where you never had a permission to begin with. Windows ACLs can subtract permissions you’ve already gained with an explicit deny.
NFS 4 has a simplified network model, and only needs the single port available.
- tcp/udp 2049
Defining exports on a Linux NFS server
In a normal Linux NFS server, exports are defined in
I won’t be able to explain every possible option, but there is a basic file format that this file follows. I’m going to begin by putting the best documentation I could find here:
- 📖 21.7. The /etc/exports Configuration File Red Hat Enterprise Linux 5 | Red Hat Customer Portal
- 📖 8.6. Configuring the NFS Server Red Hat Enterprise Linux 7 | Red Hat Customer Portal
The general format of
/etc/exports is as follows:
/path/to/share address(options) [address2(options)] # for example: /srv/nfs 192.168.1.0/24(rw,sync,no_root_squash) 192.168.101.0/24(ro,root_squash)
Some common options are:
rwto decide whether the share is writable
no_root_squashto decide whether the client is allowed to exercise root privileges
If the server is not configured to strip the root user of rights, the root user can do whatever they want to in NFS3. Note that it is most common for the NFS server to strip the root user’s rights and assert that root is nobody special.
Windows, Linux and MacOS
ID mapping is the most difficult problem to solve when integrating Windows, Linux, and MacOS clients.
Generally speaking, for all of it to work correctly, all clients and servers will need to agree on what USER ID (uid) and GROUP ID (gid) each named security principal should have. A security principal is any object that can be added to an ACL - User accounts, computer accounts, and windows security groups (not to be confused with mail groups, which are not always security groups but sometimes are) Windows uses a SID - which looks like
S-1-5-21-11111111-222222222-3333333-500. All POSIX systems (Linux, MacOS, BSD, Solaris, etc) use plain integers like 123456783 to represent users and groups. In a Linux system, all groups are security principals, there is no concept of a non-security-principal list of users on a normal Linux system.
Since in my case the users who need to access the data are all defined in Active Directory - that is our source of truth. So we need some way to generate a user ID for all of our users that all of our clients and servers will respect. For our Linux servers and clients, there is a built in solution. We’ll join those to Active Directory, and rely on sssd to generate an identifier.
Since sssd generates this number the same way every time based on the end section of the user’s sid (500 in our example earlier), we can be reasonably sure that in our very simple, single forest & single domain setup, all Linux clients will agree on any given user’s UID, and we can use that to set our ACLs. If SSSD is joined to more than one domain, you may see different results, as I’m not an expert on how SSSD determines how each domain’s prefix is determined, and I couldn’t find any good explanation on how it works, and I can’t read C code all that well.
In our example above, it appears the scheme is [4 characters of domain prefix determined through math]+[zero-padded 6 characters using the last section of the domain SID]. In the case of the domain administrator’s account (
S-1-5-21-11111111-222222222-3333333-500 ) the uid is something like
Note that adding any system to a Windows Active Directory domain also establishes a Kerberos trust between the client and the domain, but we won’t cover that a lot here. This is important for SMB access or NFS with Kerberos auth.
Identity mapping in Isilon
In this case, there is an added wrinkle - Isilon. Isilon runs a heavily modified BSD operating system. Isilon can either generate an ID for each user and group on its own - which might be fine if all our clients were using SMB only - or you can explicitly define what each user and group’s ID numbers should be in Active Directory.
In our case since we have a mix of client operating systems and protocols accessing this same dataset, it is important to make sure everything involved agrees on what those numbers should be.
Creating a consistent mapping
For Linux clients, sssd can be configured to read the id numbers out of AD, but we do not go out of our way to make this happen. This being the case, sssd is always allowed to generate an id based on the sid of the user or group. Since we use AUTH_SYS ( known in the Isilon user interface as UNIX (system) ), the clients still pass a user ID number to the NFS server (Isilon), and permissions are evaluated based on that - it is important that the nfs clients pass a number the Isilon NFS server will recognize.
If we do not specifically define a uid or gid on the AD objects, the Isilon will generate a number based on it’s own chosen math to represent the user. This will not match the id that SSSD will generate, so we are forced to fill in the uidNumber 📖 uidNumber attribute - Win32 apps and gidNumber 📖 gidNumber attribute - Win32 apps for each user or group with a number that matches what SSSD would use.
We can get this number in one of two ways - wing it based on the
[4 characters of domain prefix determined through math]+[zero-padded 6 characters using the last section of the domain SID] guess at the formula we’ve made, or we can simply create the AD object and run getent passwd <username> for users or getent group <group name> (using quotes if the group name contains spaces) for groups from any Linux machine joined to AD, then write that value to active directory on the object in question.
Linux client & Isilon additional setup
Linux has an
idmapd service - it’s job is to handle .. id mapping tasks.
/etc/idmapd.conf, which is an INI file - find the
[General] section and make sure it contains an entry similar to this:
Domain = AD.EXAMPLE.COM
It should match your AD domain, in uppercase, and the Isilon should also be configured with the same name in the web interface under Protocols → “UNIX sharing (NFS)” → <your access zone> → Zone settings. the domain name goes in the field labeled “NFSv4 Domain”, also in all uppercase. I also have the following settings checked, with the help text after the setting name: Help on Protocols > UNIX Sharing (NFS) > Zone Settings. The domain name goes in the field labeled “NFSv4 Domain”, also in all uppercase. I also have the following settings checked, with the help text after the setting name: 📖 Help on Protocols > UNIX Sharing (NFS) > Zone Settings
- Enable NFSv4 replace domain [Enter the NFSv4 domain for which you want to edit zone-level settings. The default is localhost.]
- Enable NFSv4 no domain UIDs [Click the check box to send UIDs/GIDs without the domain name. The default is enabled.]
- Enable NFSv4 allow numeric IDs [Click the check box send owners/groups as UIDs/GIDs. The default is disabled when lookups fail or if NFSv4 No Names is enabled.]
I do not pretend to know how many of those are actually necessary, I just know that this arrangement seems to work.
In addition, under Protocols → “UNIX sharing (NFS)” → <your access zone> → Export Settings, I have “Map lookup UID” set to “Yes”. 📖 Help on Protocols > UNIX Sharing (NFS) > NFS Exports
Troubleshooting access from Isilon on a client
Squash / ACL
If you expect to be able to get into a folder and you are running as root or any user local to a client, there are a few things that might be getting in your way:
- The export may have
- There is no ACL that gives you access
Being root is no guarantee of access from an NFS client
You can instead impersonate the user in question which is easy enough if you’re using AUTH_SYSTEM - the root user can
su to any other user and simulate the same access patterns to see if the ACLs are working properly. If you’re using Kerberos auth, this is not guaranteed to work since you won’t have the proper kerberos ticket for that user account.
Verify ID Mapping
- From any Linux client, run
getent user <username>and take note of the value.
- From the Isilon web console, choose the Access menu, then Membership and roles.
- Select the correct zone from the “Current access zone” drop-down menu.
- On the User mapping tab, click browse and find the user or type the correct
DOMAIN\usernamevalue, and click “Test mapping”
- You will get a listing of the user’s SID and UID number, along with the full listing of each group they’re a member of, including their GIDs and SIDs.
- If the UID or any relevant group GID does not match the output from SSSD, then the ad object does not have it’s attributes set properly. You will need to open Active Directory Users and Computers, ensure View → Advanced Features is checked, browse to the object, and open it. Using the Attribute Editor tab, you can verify whether the uidNumber/gidNumber attribute is blank or disagrees with the value from SSSD. If so, fill in the value you obtained from SSSD.
Do not search for the account and try to edit the properties from the search result - the attribute editor tab does not always appear when the properties are viewed this way
If you expect to be able to get into a folder and you are running as root or any user local to a machine - even if it’s created through a configuration management tool and exists on many machines, it is almost certain that there is no ACL that grants you access - and being root is no guarantee of access.
The most direct approach to verifying an ACL if the same dataset is shared over SMB is to browse to the file or folder in question in Windows' file explorer, and viewing its properties. If the user is in a group or a named user ACL, they should have whatever access is listed there. There could be occasions where the process to set the ACLs broke part-way through and did not reach the level of the file system where access is needed, in which case it may be necessary to re-apply them.
There are other methods you can use to view the ACLs, and they may be occasionally useful.
From the Isilon command line:
cd /ifs/<path>/<to>/<folder> ls -led .
From and NFS4 client:
cd /<path>/<to>/<mount point> nfs4_getfacl .
If all of the ID mapping is set correctly on the client and the Isilon, the ACLs will appear with the correct domain name, etc. If not, revisit the “Linux client & Isilon additional setup” section above.
I hope this has been helpful to you.