Disable nscd caching #50316

arianvp · 2018-11-13T16:11:03Z

Motivation for this change

Nscd caching was interferring with systemd's nss module. Disable caching as
caching is not the reason why we use nscd. We use it to resolve nss modules correctly. See commit message for more
detailed information.

Fixes #50273

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

arianvp · 2018-11-13T16:20:36Z

Supersedes #50042 if we also disable host caching

arianvp · 2018-11-13T16:23:01Z

CC @Mic92 . We could also replace it with unscd while we're at it.

arianvp · 2018-11-13T16:27:35Z

I haven't actually tried setting enable-cache no but just took @dezgeg 's word for it that it only works if we set ttls to 0 . We do the same when we enable sssd. I also elaborated this in the commit message but haven't actually tested this hypothesis. I could also try fully disabling the caches.

Mic92 · 2018-11-13T16:30:13Z

I think we should first disable negative-time-to-live caches for hosts:

negative-time-to-live hosts 0

This should fix problems people have when they cannot reach a site, when the hostname was resolved while having no connectivity.
And then test unscd for a while. It could have even funnier bugs.

lheckemann · 2018-11-13T16:31:43Z

I think this is worth mentioning in the release notes.

peterhoeg · 2018-11-14T01:01:02Z

I think we should first disable negative-time-to-live caches

We might as well disable the positive TTL too and say that nscd is not for caching. Whenever something unexplained happens with name resolution, my first step is always just to restart nscd which solves it nine times out of 10.

edolstra · 2018-11-14T09:26:14Z

Hm, what's the advantage of disabling caching of positive results?

arianvp · 2018-11-14T09:39:30Z

Basically I wanted to be 100% sure nothing fishy is happening. I disabled everything to be 100% sure no funny stuff is happening. Systemd project acknowledged that they are not nscd-aware and they might look into fixing that. This means there might be more subtle bugs lurking in the presence of caching. I'm not sure if the positive cache is invalidated when a systemd unit with Dynamic users stops. If it doesn't, this might cause funny bugs when another systemd service with Dynamic users recycles that uid and discovers there's already a name associated to the uid.

user lookups are basically instantaneous and the only moment we would want to cache them is in the presence of LDAP to improve performance. But sssd already does its own caching and so we already disable nscd caching when sssd is enabled in NixOS and Red Hat (https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/usingnscd-sssd)

So the only moment we seem to have caching is when the passwd database is backed by a file. I think this is a bit overkill and will cause more issues than gains.

Edit: @edolstra oh I think I now just realised you were talking about caching of hosts, not the other caches.

arianvp · 2018-11-14T11:21:56Z

I've disabled negative caching for hosts for now. and left positive caching in.

I'm not very familiar with the release process of NixOS. What should a release notes entry look like, and where do I put it?

Mic92 · 2018-11-14T11:27:00Z

You can take the last release as an example:

https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-1809.xml

your one should go here:

https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-1903.xml

arianvp · 2018-11-14T11:45:16Z

Now that sssd and nscd ship basically the same config, is it worth to remove the settings from sssd ?

Mic92 · 2018-11-14T17:21:28Z

@arianvp when they are 100% the same, we can drop the sssd one.

Mic92 · 2018-11-14T17:22:54Z

nixos/modules/services/system/nscd.conf

 suggested-size          group           211
 check-files             group           yes
 persistent              group           no
 shared                  group           yes

 enable-cache            hosts           yes
 positive-time-to-live   hosts           600
-negative-time-to-live   hosts           5
+negative-time-to-live   hosts           0


I wonder if we at least should backport this one, since it can be quiet annoying if websites do not load correctly after connecting to a hotspot.

Yes, I’ve had this problem for more than a month now, so the release must have it as well.

peterhoeg · 2018-11-15T03:10:29Z

Hm, what's the advantage of disabling caching of positive results?

We don't gain anything by having it enabled and disabling it removes one potential source of problems.

jokogr · 2018-11-15T08:36:48Z

nixos/doc/manual/release-notes/rl-1903.xml

+   <listitem>
+    <para>
+      The <literal>nscd</literal> now disables all caching of
+      <literal>passwd</literal> and <literal>group</litral> databases by


Should be </literal> the closing tag in group

nix build -f my-nixpkgs/nixos config.system.build.manual --arg configuration {} to make sure the manual builds :)

Mic92 · 2018-11-15T09:45:37Z

@peterhoeg we gain a dns cache if it is enabled. I don't see how this would cause problems if it caches valid entries. DNS is designed around for those caches and uses TTLs itself to control invalidation.

arianvp · 2018-12-05T13:23:22Z

@edolstra I reverted your revert, as I don't think it fixes anything. I'm 100% sure that nscd already invalidates the cache when /etc/resolv.conf is changed, through inotify.

arianvp · 2018-12-05T13:39:47Z

After reading the discussion at #42569 perhaps we should only unrevert the commit Eelco did, after we fully disable caching in nscd. I added Eelco's revert to the patchset again ...

I think we should merge this as is, and then in a next PR:

introducde systemd-resolved
Disable positive caching of domains in nscd
Remove all forms of restart triggers / cache invalidations of nscd

flokli · 2018-12-06T15:55:14Z

nixos/doc/manual/release-notes/rl-1903.xml

+     <literal>libnss_systemd.so</literal> module which is used by
+     <literal>systemd</literal> to manage uids and usernames in the presence
+     of <literal>DynamicUser=</literal> in systemd services.
+     The was already the default behaviour in presence of


Suggested change

The was already the default behaviour in presence of

This was already the default behaviour in presence of

flokli

more doc feedback

arianvp · 2018-12-12T13:55:16Z

I have addressed the doc issues. I think this is ready for merge. I have created a follow-up ticket with the rest of the tasks that popped up in this discussion #51911

flokli · 2018-12-12T14:22:29Z

nixos/modules/services/system/nscd.conf

@@ -4,25 +4,41 @@ paranoia                no
 debug-level             0

 enable-cache            passwd          yes


do we want to disable caches where we set a zero ttl for both positive and negative? Seems less confusing to me :-)

As eelco mentioned, this would have a serious performance penalty. so I don't want to change it until #51911 is implemented. I think this is a good compromise where people do not get failed lookups when switching networks, but do also have performance. Until we figure out the resolved business

Oh I misread: See the original commit message why we can't do this:

Note that we can not just put in /etc/nscd.conf: enable-cache passwd no As this will actually cause glibc to _not_ forward the call to nscd at all, and thus never reach the nss modules. Instead we set the negative and positive cache ttls to 0 seconds as a workaround. This way, Glibc will always forward requests to nscd, but results will never be cached.

I added this as a comment to nscd.conf to clarify

Systemd provides an option for allocating DynamicUsers which we want to use in NixOS to harden service configuration. However, we discovered that the user wasn't allocated properly for services. After some digging this turned out to be, of course, a cache inconsistency problem. When a DynamicUser creation is performed, Systemd check beforehand whether the requested user already exists statically. If it does, it bails out. If it doesn't, systemd continues with allocating the user. However, by checking whether the user exists, nscd will store the fact that the user does not exist in it's negative cache. When the service tries to lookup what user is associated to its uid (By calling whoami, for example), it will try to consult libnss_systemd.so However this will read from the cache and tell report that the user doesn't exist, and thus will return that there is no user associated with the uid. It will continue to do so for the cache duration time. If the service doesn't immediately looks up its username, this bug is not triggered, as the cache will be invalidated around this time. However, if the service is quick enough, it might end up in a situation where it's incorrectly reported that the user doesn't exist. Preferably, we would not be using nscd at all. But we need to use it because glibc reads nss modules from /etc/nsswitch.conf by looking relative to the global LD_LIBRARY_PATH. Because LD_LIBRARY_PATH is not set globally (as that would lead to impurities and ABI issues), glibc will fail to find any nss modules. Instead, as a hack, we start up nscd with LD_LIBRARY_PATH set for only that service. Glibc will forward all nss syscalls to nscd, which will then respect the LD_LIBRARY_PATH and only read from locations specified in the NixOS config. we can load nss modules in a pure fashion. However, I think by accident, we just copied over the default settings of nscd, which actually caches user and group lookups. We already disable this when sssd is enabled, as this interferes with the correct working of libnss_sss.so as it already does its own caching of LDAP requests. (See https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/usingnscd-sssd) Because nscd caching is now also interferring with libnss_systemd.so and probably also with other nsss modules, lets just pre-emptively disable caching for now for all options related to users and groups, but keep it for caching hosts ans services lookups. Note that we can not just put in /etc/nscd.conf: enable-cache passwd no As this will actually cause glibc to _not_ forward the call to nscd at all, and thus never reach the nss modules. Instead we set the negative and positive cache ttls to 0 seconds as a workaround. This way, Glibc will always forward requests to nscd, but results will never be cached. Fixes NixOS#50273

Hopefully fixes NixOS#50290

It was the last database that wasn't listed.

flokli · 2018-12-12T14:48:43Z

LGTM.

Mic92 · 2018-12-12T23:18:12Z

I backported @edolstra revert in 25d0880, which fix the biggest pain point we had at the moment. Before backporting a change like this one, I would give a test period on unstable. We might not need to backport at all since the problems with DynamicUser should not affect many people.

flokli · 2019-02-04T17:40:24Z

@arianvp Had a discussion at FOSDEM about how we (ab)use nscd to pass nss modules, as described in 1d5f4cb.

This seems to be a bad idea, due to glibc falling back to local resolving (with broken nss modules) after very quick timeouts.

Can we get glibc to load nss modules from some global /run/nss-modules path instead, pointing to something in the nix-store?

Mic92 · 2019-02-04T17:46:36Z

@flokli This can break if a program was linked against a different version of glibc
since. This is why we use nscd in the first place.

arianvp · 2019-02-05T14:31:12Z

Only in the case where glibc breaks abi right? We do something similar for OpenGL/Vulkan iirc

flokli · 2019-02-05T16:27:49Z

I opened #55276 about that.

Mic92 · 2019-02-05T16:42:51Z

@arianvp This could make it impossible to mix releases, but not just for OpenGL applications but every application.

This effectively disables nscd's built-in hosts cache, which turns out to be erratic in some cases. We only use nscd these days as a more ABI-neutral NSS dispatcher mechanism. Local caching should still be possible with local resolvers in /etc/resolv.conf (via the `dns` NSS module), or without local resolvers via systemd-networkd (via the `resolve` nss module) We don't set enable-cache to no due to NixOS#50316 (comment).

arianvp requested a review from infinisil as a code owner November 13, 2018 16:11

arianvp force-pushed the fix-dynamic-user branch 3 times, most recently from 67b7153 to f667a18 Compare November 13, 2018 16:15

arianvp mentioned this pull request Nov 13, 2018

Weird DNS resolutions with NixOS + NetworkManager #50290

Closed

arianvp changed the title ~~Fix dynamic user~~ Disable nscd caching Nov 13, 2018

GrahamcOfBorg added 6.topic: nixos 8.has: module (update) labels Nov 13, 2018

Mic92 mentioned this pull request Nov 13, 2018

nscd: disable by default #50042

Closed

8 tasks

GrahamcOfBorg added 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Nov 13, 2018

GrahamcOfBorg added 8.has: changelog 8.has: documentation 2.status: merge conflict labels Nov 14, 2018

Mic92 reviewed Nov 14, 2018

View reviewed changes

jokogr reviewed Nov 15, 2018

View reviewed changes

arianvp force-pushed the fix-dynamic-user branch from b4e2fbd to aeb4aa1 Compare December 5, 2018 13:38

flokli mentioned this pull request Dec 6, 2018

GCE OSLogin module: init #51566

Merged

14 tasks

flokli reviewed Dec 6, 2018

View reviewed changes

flokli requested changes Dec 6, 2018

View reviewed changes

arianvp mentioned this pull request Dec 12, 2018

Replace nscd caching with systemd-resolved #51911

Closed

flokli reviewed Dec 12, 2018

View reviewed changes

arianvp added 8 commits December 12, 2018 15:35

nixos/systemd: Add a regression test for NixOS#50273

eb88005

nixos/nscd: Disable negative caching of hosts

99d3279

Hopefully fixes NixOS#50290

nixos/nscd: Add release note entry about nscd changes

335b41b

nixos/nscd: Merge nscd and sssd-nscd config

de76c16

nixos/nscd: also add netgroup to the config

a74619c

It was the last database that wasn't listed.

nixos/nscd: Address doc feedback

ef6ed03

nixos/nscd: Add a descriptive comment to the nscd configuration

1d5f4cb

arianvp force-pushed the fix-dynamic-user branch from 702a21e to 1d5f4cb Compare December 12, 2018 14:36

Mic92 merged commit 5feba45 into NixOS:master Dec 12, 2018

flokli mentioned this pull request Dec 14, 2018

nslcd: restart when nslcd.conf changes and let pwdutils tools use libnss_ldap.so #46792

Closed

9 tasks

flokli mentioned this pull request Feb 5, 2019

provide nss modules globally instead of hacking them with nscd #55276

Closed

flokli mentioned this pull request May 31, 2020

nixos/nscd: set positive hosts caching ttl to 0 #89274

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disable nscd caching #50316

Disable nscd caching #50316

arianvp commented Nov 13, 2018 •

edited

arianvp commented Nov 13, 2018

arianvp commented Nov 13, 2018

arianvp commented Nov 13, 2018 •

edited

Mic92 commented Nov 13, 2018 •

edited

lheckemann commented Nov 13, 2018

peterhoeg commented Nov 14, 2018

edolstra commented Nov 14, 2018

arianvp commented Nov 14, 2018 •

edited

arianvp commented Nov 14, 2018

Mic92 commented Nov 14, 2018

arianvp commented Nov 14, 2018

Mic92 commented Nov 14, 2018

Mic92 Nov 14, 2018 •

edited

Profpatsch Nov 20, 2018

peterhoeg commented Nov 15, 2018

jokogr Nov 15, 2018

lheckemann Nov 15, 2018

Mic92 commented Nov 15, 2018

arianvp commented Dec 5, 2018

arianvp commented Dec 5, 2018 •

edited

flokli Dec 6, 2018

flokli left a comment

arianvp commented Dec 12, 2018

flokli Dec 12, 2018

arianvp Dec 12, 2018

arianvp Dec 12, 2018 •

edited

arianvp Dec 12, 2018

flokli commented Dec 12, 2018

Mic92 commented Dec 12, 2018 •

edited

flokli commented Feb 4, 2019

Mic92 commented Feb 4, 2019

arianvp commented Feb 5, 2019

flokli commented Feb 5, 2019

Mic92 commented Feb 5, 2019

	The was already the default behaviour in presence of
	This was already the default behaviour in presence of

		@@ -4,25 +4,41 @@ paranoia no
		debug-level 0

		enable-cache passwd yes

Disable nscd caching #50316

Disable nscd caching #50316

Conversation

arianvp commented Nov 13, 2018 • edited

Motivation for this change

Things done

arianvp commented Nov 13, 2018

arianvp commented Nov 13, 2018

arianvp commented Nov 13, 2018 • edited

Mic92 commented Nov 13, 2018 • edited

lheckemann commented Nov 13, 2018

peterhoeg commented Nov 14, 2018

edolstra commented Nov 14, 2018

arianvp commented Nov 14, 2018 • edited

arianvp commented Nov 14, 2018

Mic92 commented Nov 14, 2018

arianvp commented Nov 14, 2018

Mic92 commented Nov 14, 2018

Mic92 Nov 14, 2018 • edited

Choose a reason for hiding this comment

Profpatsch Nov 20, 2018

Choose a reason for hiding this comment

peterhoeg commented Nov 15, 2018

jokogr Nov 15, 2018

Choose a reason for hiding this comment

lheckemann Nov 15, 2018

Choose a reason for hiding this comment

Mic92 commented Nov 15, 2018

arianvp commented Dec 5, 2018

arianvp commented Dec 5, 2018 • edited

flokli Dec 6, 2018

Choose a reason for hiding this comment

flokli left a comment

Choose a reason for hiding this comment

arianvp commented Dec 12, 2018

flokli Dec 12, 2018

Choose a reason for hiding this comment

arianvp Dec 12, 2018

Choose a reason for hiding this comment

arianvp Dec 12, 2018 • edited

Choose a reason for hiding this comment

arianvp Dec 12, 2018

Choose a reason for hiding this comment

flokli commented Dec 12, 2018

Mic92 commented Dec 12, 2018 • edited

flokli commented Feb 4, 2019

Mic92 commented Feb 4, 2019

arianvp commented Feb 5, 2019

flokli commented Feb 5, 2019

Mic92 commented Feb 5, 2019

arianvp commented Nov 13, 2018 •

edited

arianvp commented Nov 13, 2018 •

edited

Mic92 commented Nov 13, 2018 •

edited

arianvp commented Nov 14, 2018 •

edited

Mic92 Nov 14, 2018 •

edited

arianvp commented Dec 5, 2018 •

edited

arianvp Dec 12, 2018 •

edited

Mic92 commented Dec 12, 2018 •

edited