Google LDAP does not allow filtering by uidNumber by default causing SSSD cache refreshes to fail

[GlitchWitch]

Problem

When most Linux applications try to authenticate a user against PAM, they query the user by the numerical ID ("uidNumber") of the user account. Under most circumstances, when SSSD performs the lookup for an LDAP user by uidNumber it queries it's local cache which stores the uidNumber as an alias for a named domain account, therefore correctly identifying a user profile by the uidNumber.

However, in cases where the cache entry is expired or missing, SSSD will query the backend LDAP server by the user's uidNumber to refresh the user profile. Unfortunately, Google's LDAP service does not allow querying by uidNumber by default, and we have not found a simple way to enable this behavior for all users (see Workaround 1 below). This causes the LDAP query to return zero matches, leaving the user unable to authenticate. Until resolved, this causes issues including new shell sessions being unable to identify the user, broken file permissions, Snap applications fail to open including clicking links to open in browser, among other issues.

This issue occurs very frequently, usually several times per day, making it very disruptive. It is easy to know when this issue has occurred by checking if bash identifies the user name as "I have no name!" when opening a new terminal, potentially also accompanied by a group ID lookup failing:

groups: cannot find name for group ID 706618022
I have no name!@glitchsecure-laptop-383002X:~$ 

Steps to Reproduce

1. Configure a Google Workspace domain with the Google Secure LDAP provider using digital certificates for authentication
2. Configure a Linux device to use SSSD with LDAP authentication (we use Ubuntu 24.04). It is best not to configure an extended cache timeout to reproduce the issue quickly
3. Wait for an authentication lookup to occur after the cached LDAP profile has expired.

---

Alternatively, to simply test that it is not possible to perform a query filtering on uidNumber against Google LDAP without setting up a device, you can replicate the query and filter used by SSSD manually with ldapsearch:

export LDAPTLS_CERT="/path/to/LDAP.crt" LDAPTLS_KEY="/path/to/LDAP.key"
# the exact query used by SSSD
ldapsearch -H ldaps://ldap.google.com:636 -b "dc=glitchsecure,dc=com" '(&(uidNumber=[uidNumber])(objectclass=posixAccount)(uid=*)(&(uidNumber=*)(!(uidNumber=0))))'
# simplified query with the same problem
ldapsearch -H ldaps://ldap.google.com:636 -b "dc=glitchsecure,dc=com" '(&(uidNumber=[uidNumber])(objectclass=posixAccount))'

Export the correct paths to your LDAP certificate and key. Make sure to replace dc=glitchsecure,dc=com with your domain, and [uidNumber] with the Posix account ID in the ldapsearch commands.

ldap query results with uidNumber	ldap query results with username

Expected Behavior

The query by uidNumber should return the relevant user profile. See cache-expired-success.txt in the Logs section below for an example successful query by SSSD.

Actual Behavior

The query by uidNumber returns 0 results, causing authentication to fail until the profile is manually refreshed (Workaround 2). See cache-expired-failed.txt in the Logs section below for an example failed query by SSSD.

Known Workarounds

1. Google allows querying by uidNumber if an administrator uses the Google Admin SDK to set a "posixAccounts" attribute with the correct uid and gid (source), and with systemId set to a blank string (source). Doing this does resolve the error and allow uidNumber lookups to succeed.
   • We have not identified a way to simply and securely automate this process. It is not possible to use the LDAPS protocol from within a Google App script, and we cannot find an existing HTTP API to fetch Google's LDAP profile information. Without that, we would need to run an external automation that has the ability to use the LDAPS protocol directly to get account UIDs, or insecurely trust a user to pass their UID to a Google Web App Script POST endpoint to set the information on their domain profile.
   • Updating profile attributes requires an admin level scope, so we can't feasibly allow users to update this information themselves.
   • It is worth noting that when performing an LDAP search by name manually (eg. with ldapsearch), the Google LDAP service does return uidNumber and gidNumber regardless of if the posixAccounts attribute is set on the domain user. So we know that Google is storing the information somewhere, we just don't know if we can access it without the LDAPS protocol.
2. It is possible to trigger a user lookup by account name instead of uidNumber. The easiest ways to do this are to lock your session and sign back in, or run the command id [name] with your domain account name, eg. id noah.curoe. Doing this performs a query against the LDAP backend using the account name which Google does allow filtering by. Afterwards, the LDAP cache will be repopulated with this profile including the uidNumber alias, allowing all authentication to work again until the cache entry next expires.
   • While this workaround is easy and effective, it is not preventative and requires manual action to be taken after the issue has already occurred causing workflow disruptions.
3. Extending the cache timeout duration in the SSSD configuration may reduce the frequency of this issue by reducing how often SSSD performs a uidNumber query against the LDAP backend.
   • Since we are a remote first company, it is normal for a user to leave their laptop running for potentially weeks at a time well beyond any reasonable cache expiry timeout.
   • Extending the cache timeout is also a potential security issue as it delays syncing updates between the LDAP backend and the device.

The above workarounds are either only temporarily effective, insecure, or require manual action to be taken to correct the issue and are therefore not ideal.

Desired Solution

Since the default configurations provided by Google result in this issue, and we have not found a way to automate setting posixAccounts information in Google easily and securely, we propose the following changes to SSSD if Google is not able to resolve this.

Ideally, we would want to configure SSSD to always refresh the LDAP cache using the account name instead of the uidNumber. The account name could be sourced from the expired cache entry for the cache refresh query instead of using the uidNumber. While SSSD allows modifying the search filters used for user lookup by changing the filter base, it does not seem possible to pass in the login name of the user or the account name from the LDAP cache.

References

Google Documentation

• This Google help article details the requirement to set the posixAccounts attribute on each individual user to enable querying LDAP profiles by uidNumber.
• This Google Admin SDK documentation section details about the posixAccounts attribute and how to set it using the API.

Logs

SSSD logs with debug level set to 8 are attached which details the different behaviors of the LDAP lookup process in different scenarios which helped us diagnose the issue.

• cache-hit.txt shows an SSSD log where sudo queried authentication by uidNumber. In this case since the profile was in the cache and not expired, SSSD correctly translated the uidNumber to the user profile without querying the backend server, allowing authentication to succeed.
• cache-expired-failed.txt shows an SSSD log where bash tries to lookup a user by uidNumber. While SSSD found the user in cache, it was expired and queried the LDAP backend to refresh the authentication. The query by uidNumber returned no results, so the authentication failed.
• name-lookup.txt shows an SSSD log triggered by running id noah.curoe after authentication has failed. This triggers a query against the LDAP backend by name ("uid") instead of uidNumber, which returns the correct profile and successfully updates the cache with the refreshed profile. After this authentication worked until the cache next expired.
• cache-expired-success.txt shows an abbreviated SSSD log after setting the posixAccounts attribute in Google. This log shows SSSD identifying that the LDAP profile in cache has expired, but in this case the query against the Google LDAP provider using the uidNumber successfully returned the correct profile allowing the cache to refresh correctly, leaving authentication working as normal.

Special thanks to @noahc3 on our team for testing and largely compiling this information.

[alexey-tikhonov]

Unfortunately, Google's LDAP service does not allow querying by uidNumber

What does Google support say about this?

[GlitchWitch]

@alexey-tikhonov as noted in the issue, this Google help article details the requirement to set the posixAccounts attribute on each individual user to enable querying LDAP profiles by uidNumber.

Note: You won't be able to search for a user using their uidNumber or gidNumber unless these attributes were set by an admin using the Admin SDK API. When these attributes are set using the API, the posixAccount systemId attribute must also be set or the uidNumber and gidNumber will not be searchable.

See the notes for Workaround 1 for why doing this for all users isn't feasible.

[GlitchWitch]

Potentially related/duplicate issue; #7299

[sumit-bose]

Hi,

thank you for the detailed report. Although I think that it would be good if Google can fix this behavior I think your request is valid and having something like a preferred search attribute is a good idea. I'm currently working on a generic IdP support for SSSD where this would come handy. But even with existing LDAP environments some attributes might be more suitable than others e.g. because of indexing (my guess is that indexing might be the reason for the odd behavior of Google's LDAP service, maybe this is even mentioned in one of the documents you have linked but I have to admit I haven't read all of them).

I'm currently thinking about where would be the best place to implement this. A straight forward way to handle yout request would be the cache lookup in the NSS responder because here we have already read the cached entry and can take e.g. the user name to create the refresh request for the backend. But SSSD can handle multiple backends and not all might need this kind of translation. This means the backend has to play a role here as well. So maybe it can add a new attribute the the cached user and group attribute with the preferred refresh type which is then evaluated by the NSS responder.

bye,
Sumit

[GlitchWitch]

Hey Sumit,

Thank you for your response and review!

Although I think that it would be good if Google can fix this behavior I think your request is valid and having something like a preferred search attribute is a good idea.

I definitely agree that ideally the fix can and should be performed by Google itself, especially since default configurations of their "Secure LDAP" fail to respond to queries with the UID which is known to already be stored by them and is the underlying cause for the breakage. Sadly our attempts to escalate the issue through Google workspace support so far seem futile and met with misunderstanding. We even got sent back to our own issue listing here 🫠.

Glad to hear the suggestion for a preferred search attribute was welcome and potentially useful elsewhere. I am hopeful that might be a more permanent workaround in lieu of Google properly addressing this at a product level.

I'm currently thinking about where would be the best place to implement this.

Do let us know if there is anything we can do to further help here. We are more than happy to test this against Google Workspace's Secure LDAP once a PR is ready.