nix-daemon: Add CPU and IO weight options #26520

Baughn · 2017-06-11T19:08:18Z

Motivation for this change

Making Chrome (or other UI elements) less laggy.

CPU prioritization by way of 'nice' doesn't work very well. Modern systems have a better option, in that systemd can be used to set CPU timeslice weighting explicitly. This requires turning on accounting for the same, globally across the system, which I do with the systemd config patch.

The second patch makes the CPU weighting configurable for the nix-daemon service, and defaults it to the minimum possible. I believe this is a good default for desktop systems, as well as most servers; the builds are generally batch processes, and this setting does not prevent it from using all idle time.

All of the above is duplicated for I/O accounting, although I cannot test the impact from this due to running on NVMe.

There is a very slight performance impact from the accounting, but not so you'd notice. My experience is that it (if it exists at all) is vastly outweighed by usability improvements through making interactive sessions higher priority.

(I've been told that CPU and I/O accounting has low impact, while memory accounting has medium. This patch does not enable memory accounting.)

Other batch services should probably get the same treatment.

Things done

Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- Linux
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

mention-bot · 2017-06-11T19:08:20Z

@Baughn, thanks for your PR! By analyzing the history of the files in this pull request, we identified @edolstra, @MarcWeber and @abbradar to be potential reviewers.

grahamc · 2017-06-11T19:10:22Z

Fwiw the nix daemon should be prioritized like a user facing application, because indeed for operations likenix-shell it is a user facing. I set the process priority too low on macOS and found my nix-shell would take 5 minutes to execute, compared to just seconds before. Somewhere in the middle is probably best.

Baughn · 2017-06-11T19:17:05Z

20% by default, maybe?

I don't know about OSX, but using CPUWeight it'll still use any idle capacity. I'm not seeing any effect on shell usage, but... hmm, this is hard. Ideally it'd depend on which window is in focus.

joachifm · 2017-06-11T23:25:02Z

nixos/modules/system/boot/systemd.nix

@@ -725,6 +725,8 @@ in

      "systemd/system.conf".text = ''
        [Manager]
+        DefaultCPUAccounting=yes
+        DefaultBlockIOAccounting=yes


BlockIOAccounting refers to the legacy cgroup hierarchy (while IOWeight used above belongs to the unified hierarchy).

The way I read the manual, this is strictly redundant: enabling a type of resource control for one unit is said to imply enabling it for the slice it runs in and all parent slices.

When testing on my own system, the CPU accounting didn't seem to work without this setting. (But worked just fine even though I accidentally put it in user.conf instead of system.conf the first time, so I'm not sure what's going on.)

I can't test the IO accounting, you may well be right about that. What should I be using instead? My manual (man systemd-system.conf) just lists this one.

All the BlockIO* directives are now just IO*, so you can use DefaultIOAccounting.

The manual can explain it better: https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html#CPUAccounting=

edolstra · 2017-06-12T09:42:51Z

I feel that we should not provide per-service options that duplicate systemd resource management functionality, because then we could potentially be adding hundreds of such options. Instead users should just use the systemd options, e.g.

systemd.services.nix-daemon.serviceConfig.CPUWeight = 123;

(nix.daemonNiceLevel is a legacy option that predates systemd, necessary because we had no such mechanism for Upstart jobs.)

I have no strong feelings about turning on accounting by default, but if it's not the upstream default, then maybe we shouldn't.

BTW, according to the systemd docs, CPUWeight and IOWeight require the unified control group hierarchy. But, AFAIK, we don't enable that yet. (But maybe it works in the hybrid mode, which is the systemd 233 default, I think.)

Mic92 · 2017-06-12T14:30:39Z

If CPUWeight is enabled for one service, it will be enabled for all other services as well. This would mean that enabling this in nix-daemon will basically enable it for everyone globally. At the moment CPU controller support is not possible without out-of-tree patches according to this document: https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html#CPU

Baughn · 2017-06-12T23:51:42Z

I feel that we should not provide per-service options that duplicate systemd resource management functionality, because then we could potentially be adding hundreds of such options. Instead users should just use the systemd options, e.g.

@edolstra My reasoning for adding the option is that without it, the default experience for NixOS is that the rest of the system gets laggy while nix-build is running. Which can happen as part of auto-updates.

I have no objection to using common options, but I feel that any such solution ought to have the same defaults as this patch.

joachifm · 2017-06-12T23:54:42Z

@Baughn see also previous discussion at #22672

edolstra · 2017-06-13T09:38:26Z

@joachifm Hah, I thought I commented on a similar PR recently but I couldn't find it :-)

@Baughn I don't object to setting some default values for CPUWeight etc. That's a separate issue from whether to add options.

Baughn · 2017-06-15T15:26:28Z

@edolstra So let's talk interfaces.

The reason I'd like to add the options is so people will know they exist, and because there already is a niceness option there. With reasonable defaults that becomes less important, but my first thought on seeing "Huh, nix-build is eating all my CPU time" was to look for options relating to nix-build, not systemd.

On the other hand, long-term it would be good if systemd is the first place people look. It just isn't the case yet, since we haven't had these tools for very long.

How would you feel if, as a compromise:

I still add the option, with the same 20% default as in the current patch, but as an alias to the direct systemd option. I think it's too early to expect people to otherwise find it.
Optionally, I can deprecate and later remove the niceness option.

0xABAB · 2017-06-29T15:52:55Z

If you are just a user of the system and don't know anything about NixOS, you want it to run the daemon in such a way that you never notice it.

If you are a user who wants to use nix-shell, because you want to develop something and you are waiting for the answer, you want it to be done as fast as possible.

Any design which doesn't meet these two use cases is a design that isn't worth implementing.

Ekleog · 2018-10-02T01:52:37Z

(triage) @edolstra, do you have an answer for #26520 (comment) ?

Baughn · 2018-10-19T15:24:15Z

FWIW, since I upgraded to a TR4 1950X, I am no longer able to test this patch. For other reasons[1] I cannot configure Nix such that it ever uses the full CPU capacity of my machine.

[1] Specifically, the build daemon uses a ton of memory. So do compilers, etc. 32GB of memory isn't enough to gain 32-way parallelism in the builds; I have to configure it with a max of 12 threads.

infinisil · 2018-10-26T22:36:53Z

If people have a hard time knowing that they can just use systemd.* directly to customize these options, then we should make better documentation for it. We shouldn't add duplicate options for increased visibility, this only increases evaluation time for no reason.

infinisil · 2018-11-10T14:09:58Z

Closing this due to reasons mentioned above, see #26520 (comment) for how to do it without this option.

Baughn force-pushed the nix-daemon branch from f342ee4 to 839af42 Compare June 11, 2017 19:11

Baughn force-pushed the nix-daemon branch from 839af42 to 03f58ba Compare June 11, 2017 23:19

joachifm reviewed Jun 11, 2017

View reviewed changes

Baughn added 2 commits June 12, 2017 03:30

systemd: Enable CPU and IO accounting by default

e051f43

nix-daemon: Add CPU and IO weight options, defaulting to very low

9975d17

Baughn force-pushed the nix-daemon branch from 03f58ba to 9975d17 Compare June 12, 2017 02:31

FRidh changed the title ~~Nix daemon~~ nix-daemon: Add CPU and IO weight options Jul 27, 2017

FRidh added the 6.topic: nixos label Jul 27, 2017

infinisil closed this Nov 10, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nix-daemon: Add CPU and IO weight options #26520

nix-daemon: Add CPU and IO weight options #26520

Baughn commented Jun 11, 2017 •

edited

mention-bot commented Jun 11, 2017

grahamc commented Jun 11, 2017

Baughn commented Jun 11, 2017

joachifm Jun 11, 2017

Baughn Jun 11, 2017 •

edited

joachifm Jun 11, 2017

joachifm Jun 11, 2017

edolstra commented Jun 12, 2017 •

edited

Mic92 commented Jun 12, 2017 •

edited

Baughn commented Jun 12, 2017

joachifm commented Jun 12, 2017

edolstra commented Jun 13, 2017

Baughn commented Jun 15, 2017 •

edited

0xABAB commented Jun 29, 2017 •

edited

Ekleog commented Oct 2, 2018

Baughn commented Oct 19, 2018

infinisil commented Oct 26, 2018

infinisil commented Nov 10, 2018

nix-daemon: Add CPU and IO weight options #26520

nix-daemon: Add CPU and IO weight options #26520

Conversation

Baughn commented Jun 11, 2017 • edited

Motivation for this change

Things done

mention-bot commented Jun 11, 2017

grahamc commented Jun 11, 2017

Baughn commented Jun 11, 2017

joachifm Jun 11, 2017

Choose a reason for hiding this comment

Baughn Jun 11, 2017 • edited

Choose a reason for hiding this comment

joachifm Jun 11, 2017

Choose a reason for hiding this comment

joachifm Jun 11, 2017

Choose a reason for hiding this comment

edolstra commented Jun 12, 2017 • edited

Mic92 commented Jun 12, 2017 • edited

Baughn commented Jun 12, 2017

joachifm commented Jun 12, 2017

edolstra commented Jun 13, 2017

Baughn commented Jun 15, 2017 • edited

0xABAB commented Jun 29, 2017 • edited

Ekleog commented Oct 2, 2018

Baughn commented Oct 19, 2018

infinisil commented Oct 26, 2018

infinisil commented Nov 10, 2018

Baughn commented Jun 11, 2017 •

edited

Baughn Jun 11, 2017 •

edited

edolstra commented Jun 12, 2017 •

edited

Mic92 commented Jun 12, 2017 •

edited

Baughn commented Jun 15, 2017 •

edited

0xABAB commented Jun 29, 2017 •

edited