Aaron

H! I’m Aaron. I keep things here that I want to remember or that I think you might find useful.

Troubleshooting NFS Access with an Isilon file server

Important things to know about NFS 3

Security Barriers

In NFS 3, there are 2 access barriers:

  • The NFS server configures exports based on network addresses, decides whether the client is allowed root access permissions, and whether an export is read+write or read-only for each client listed.
  • Basix POSIX permissions (user, group, other) - the NFS server will enforce these basic permissions, but trusts the client to not lie about what your user id / group id are in its default configuration (AUTH_SYS).

Firewall requirements

The NFS server uses several ports to make NFS 3 available:

Important things to know about NFS 4

  • NFS 4 inherits the security barriers from NFS 3 and adds access control lists - (ACLs). ACLs are more complex than the simple model and allow you more flexibility than the simple [user, group, everyone else] model that basic Linux/POSIX permissions allow. With an ACL, you layer in the basic permissions along with a list of additional users or groups and permissions. Any command that displays the ACL for an object generally also shows a representation of the basic permissions so you don’t need to run 2 commands to get a complete picture.
  • While there is such a thing as a POSIX ACL, NFS 4 ACLs follow the Windows NT permission model instead. Posix ACLs are only cumulative - you could only ever gain more permissions from them. There is no concept of a deny - except in cases where you never had a permission to begin with. Windows ACLs can subtract permissions you’ve already gained with an explicit deny.

Firewall requirements

NFS 4 has a simplified network model, and only needs the single port available.

Defining exports on a Linux NFS server

In a normal Linux NFS server, exports are defined in /etc/exports.

I won’t be able to explain every possible option, but there is a basic file format that this file follows. I’m going to begin by putting the best documentation I could find here:

The general format of /etc/exports is as follows:

/path/to/share  address(options) [address2(options)]
# for example:
/srv/nfs  192.168.1.0/24(rw,sync,no_root_squash) 192.168.101.0/24(ro,root_squash)

Some common options are:

  • ro/rw to decide whether the share is writable
  • root_squash/no_root_squash to decide whether the client is allowed to exercise root privileges

If the server is not configured to strip the root user of rights, the root user can do whatever they want to in NFS3. Note that it is most common for the NFS server to strip the root user’s rights and assert that root is nobody special.

ID Mapping

Windows, Linux and MacOS

ID mapping is the most difficult problem to solve when integrating Windows, Linux, and MacOS clients.

Generally speaking, for all of it to work correctly, all clients and servers will need to agree on what USER ID (uid) and GROUP ID (gid) each named security principal should have. A security principal is any object that can be added to an ACL - User accounts, computer accounts, and windows security groups (not to be confused with mail groups, which are not always security groups but sometimes are) Windows uses a SID - which looks like S-1-5-21-11111111-222222222-3333333-500. All POSIX systems (Linux, MacOS, BSD, Solaris, etc) use plain integers like 123456783 to represent users and groups. In a Linux system, all groups are security principals, there is no concept of a non-security-principal list of users on a normal Linux system.

Since in my case the users who need to access the data are all defined in Active Directory - that is our source of truth. So we need some way to generate a user ID for all of our users that all of our clients and servers will respect. For our Linux servers and clients, there is a built in solution. We’ll join those to Active Directory, and rely on sssd to generate an identifier.

Since sssd generates this number the same way every time based on the end section of the user’s sid (500 in our example earlier), we can be reasonably sure that in our very simple, single forest & single domain setup, all Linux clients will agree on any given user’s UID, and we can use that to set our ACLs. If SSSD is joined to more than one domain, you may see different results, as I’m not an expert on how SSSD determines how each domain’s prefix is determined, and I couldn’t find any good explanation on how it works, and I can’t read C code all that well.

In our example above, it appears the scheme is [4 characters of domain prefix determined through math]+[zero-padded 6 characters using the last section of the domain SID]. In the case of the domain administrator’s account ( S-1-5-21-11111111-222222222-3333333-500 ) the uid is something like 1234000500.

Note that adding any system to a Windows Active Directory domain also establishes a Kerberos trust between the client and the domain, but we won’t cover that a lot here. This is important for SMB access or NFS with Kerberos auth.

Identity mapping in Isilon

In this case, there is an added wrinkle - Isilon. Isilon runs a heavily modified BSD operating system. Isilon can either generate an ID for each user and group on its own - which might be fine if all our clients were using SMB only - or you can explicitly define what each user and group’s ID numbers should be in Active Directory.

In our case since we have a mix of client operating systems and protocols accessing this same dataset, it is important to make sure everything involved agrees on what those numbers should be.

Creating a consistent mapping

For Linux clients, sssd can be configured to read the id numbers out of AD, but we do not go out of our way to make this happen. This being the case, sssd is always allowed to generate an id based on the sid of the user or group. Since we use AUTH_SYS ( known in the Isilon user interface as UNIX (system) ), the clients still pass a user ID number to the NFS server (Isilon), and permissions are evaluated based on that - it is important that the nfs clients pass a number the Isilon NFS server will recognize.

If we do not specifically define a uid or gid on the AD objects, the Isilon will generate a number based on it’s own chosen math to represent the user. This will not match the id that SSSD will generate, so we are forced to fill in the uidNumber 📖 uidNumber attribute - Win32 apps and gidNumber 📖 gidNumber attribute - Win32 apps for each user or group with a number that matches what SSSD would use.

We can get this number in one of two ways - wing it based on the [4 characters of domain prefix determined through math]+[zero-padded 6 characters using the last section of the domain SID] guess at the formula we’ve made, or we can simply create the AD object and run getent passwd <username> for users or getent group <group name> (using quotes if the group name contains spaces) for groups from any Linux machine joined to AD, then write that value to active directory on the object in question.

Linux client & Isilon additional setup

Linux has an idmapd service - it’s job is to handle .. id mapping tasks.

In /etc/idmapd.conf, which is an INI file - find the [General] section and make sure it contains an entry similar to this:

Domain = AD.EXAMPLE.COM

It should match your AD domain, in uppercase, and the Isilon should also be configured with the same name in the web interface under Protocols → “UNIX sharing (NFS)” → <your access zone> → Zone settings. the domain name goes in the field labeled “NFSv4 Domain”, also in all uppercase. I also have the following settings checked, with the help text after the setting name: Help on Protocols > UNIX Sharing (NFS) > Zone Settings. The domain name goes in the field labeled “NFSv4 Domain”, also in all uppercase. I also have the following settings checked, with the help text after the setting name: 📖 Help on Protocols > UNIX Sharing (NFS) > Zone Settings

  • Enable NFSv4 replace domain [Enter the NFSv4 domain for which you want to edit zone-level settings. The default is localhost.]
  • Enable NFSv4 no domain UIDs [Click the check box to send UIDs/GIDs without the domain name. The default is enabled.]
  • Enable NFSv4 allow numeric IDs [Click the check box send owners/groups as UIDs/GIDs. The default is disabled when lookups fail or if NFSv4 No Names is enabled.]

I do not pretend to know how many of those are actually necessary, I just know that this arrangement seems to work.

In addition, under Protocols → “UNIX sharing (NFS)” → <your access zone> → Export Settings, I have “Map lookup UID” set to “Yes”. 📖 Help on Protocols > UNIX Sharing (NFS) > NFS Exports

Troubleshooting access from Isilon on a client

Squash / ACL

If you expect to be able to get into a folder and you are running as root or any user local to a client, there are a few things that might be getting in your way:

  • The export may have root_squash or all_squash defined
  • There is no ACL that gives you access

Being root is no guarantee of access from an NFS client

You can instead impersonate the user in question which is easy enough if you’re using AUTH_SYSTEM - the root user can su to any other user and simulate the same access patterns to see if the ACLs are working properly. If you’re using Kerberos auth, this is not guaranteed to work since you won’t have the proper kerberos ticket for that user account.

Verify ID Mapping

  1. From any Linux client, run getent user <username> and take note of the value.
  2. From the Isilon web console, choose the Access menu, then Membership and roles.
  3. Select the correct zone from the “Current access zone” drop-down menu.
  4. On the User mapping tab, click browse and find the user or type the correct DOMAIN\username value, and click “Test mapping”
  5. You will get a listing of the user’s SID and UID number, along with the full listing of each group they’re a member of, including their GIDs and SIDs.
  6. If the UID or any relevant group GID does not match the output from SSSD, then the ad object does not have it’s attributes set properly. You will need to open Active Directory Users and Computers, ensure View → Advanced Features is checked, browse to the object, and open it. Using the Attribute Editor tab, you can verify whether the uidNumber/gidNumber attribute is blank or disagrees with the value from SSSD. If so, fill in the value you obtained from SSSD.

Do not search for the account and try to edit the properties from the search result - the attribute editor tab does not always appear when the properties are viewed this way

Verify ACLs

If you expect to be able to get into a folder and you are running as root or any user local to a machine - even if it’s created through a configuration management tool and exists on many machines, it is almost certain that there is no ACL that grants you access - and being root is no guarantee of access.

The most direct approach to verifying an ACL if the same dataset is shared over SMB is to browse to the file or folder in question in Windows' file explorer, and viewing its properties. If the user is in a group or a named user ACL, they should have whatever access is listed there. There could be occasions where the process to set the ACLs broke part-way through and did not reach the level of the file system where access is needed, in which case it may be necessary to re-apply them.

There are other methods you can use to view the ACLs, and they may be occasionally useful.

From the Isilon command line:

cd /ifs/<path>/<to>/<folder>
ls -led .

From and NFS4 client:

cd /<path>/<to>/<mount point>
nfs4_getfacl .

If all of the ID mapping is set correctly on the client and the Isilon, the ACLs will appear with the correct domain name, etc. If not, revisit the “Linux client & Isilon additional setup” section above.

Thanks!

I hope this has been helpful to you.