Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nixos: set default for boot.tmpOnTmpfs to true #17494

Closed
wants to merge 2 commits into from

Conversation

cmfwyp
Copy link
Member

@cmfwyp cmfwyp commented Aug 3, 2016

Motivation for this change

Mounting /tmp as a tmpfs keeps /tmp clean on boot and diminishes disk usage, which in turn can save power and increase lifetime of SSD devices. It is the default behavior on Fedora, Arch Linux, and Ubuntu.

Things done

This currently uses the tmp.mount systemd unit, which has issues #14777 and #10670. It is perhaps preferable to wait for #14778 to be merged so that these issues are solved.

Since this is an incompatible change, I added an item to the release notes for 16.09.

@mention-bot
Copy link

@cmfwyp, thanks for your PR! By analyzing the annotation information on this pull request, we identified @ehmry, @domenkozar and @vcunat to be potential reviewers

@abbradar
Copy link
Member

abbradar commented Aug 8, 2016

One concern that I have is that building massive projects like Chromium can eat all the RAM with tmpfs because Nix creates build directories in /tmp. My proposed solution is to change that to /var/tmp.

@edolstra
Copy link
Member

edolstra commented Aug 8, 2016

Nitpick: Shouldn't this option be named boot.tmpfsOnTmp (as it mounts a tmpfs on /tmp, not /tmp on a tmpfs).

@cmfwyp
Copy link
Member Author

cmfwyp commented Aug 8, 2016

Yes, that would be the correct option name. I can push a commit to rename it to that and alias the old name.

@abbradar Good point. Builds are something that would preferably be done in memory, to make them faster and use the disk less, but systems may not have a swap file and therefore come out of memory pretty fast with some builds, which is a more important concern.

On the other hand, I think this would already be an issue when using Nix on most Linux distributions that mount a tmpfs on /tmp; which can mean either that this is already a problem on these distributions and that the path should be changed to /var/tmp (how likely is changing the default path for Nix to break things?), or that it has not been a problem on these distributions and as a consequence that it should not be a problem for NixOS either.

/tmp is the more semantically-correct place for builds, since they do not need to persist for a long time or even through reboots, but it is not ideal for large files.1

1: A good comparison: http://0pointer.de/blog/projects/tmp.html.

@abbradar
Copy link
Member

abbradar commented Aug 8, 2016

@cmfwyp Subjectively that would be a bigger problem on NixOS because it's the main platform for overall Nix development and also for Hydra hosts, which together (anecdotally!) account for almost all "big" builds in the Linux/Nix world.

That said, I also do like /tmp more for storing builds precisely because of its FHS semantics (they are not needed to be "persistent"). Also, speed and disk wear concerns for small builds are important too.

I'm not sure how this problem should be solved. For some prior experience, on Arch Linux /tmp is used for builds by yaourt which is a popular tool for building and installing packages from source. I needed to remount /tmp or move build to somewhere enough times for me to remember (but not really frequently) when I had 2 and later 4GB of RAM.

@romildo
Copy link
Contributor

romildo commented Aug 8, 2016

In Gentoo Linux I usually mount /tmp as a tmpfs file system, and it is used for building most of the packages. For particular packages which demand too much space I set a per package environment variable PORTAGE_TMPDIR which is used to build that package.

Maybe something in this line can be figured out for NixOS too.

This currently uses the tmp.mount systemd unit. Mounting /tmp as a
tmpfs keeps /tmp clean on boot and diminishes disk usage, which in
turn can save power and increase lifetime of SSD devices.

It is the default behavior on Arch Linux, Fedora, and Ubuntu.

Since this is an incompatible change, an item was added to the
release notes for 17.03.
@cmfwyp
Copy link
Member Author

cmfwyp commented Sep 23, 2016

The more common name seems to be "tmp on tmpfs" ([1] [2] [3] [4]), so I suggest keeping that.

Regarding large builds, I see multiple options. Adding a meta-attribute for build expressions that use large files is not an elegant solution. Build servers (Hydra or otherwise) that do builds with large files can set boot.tmpOnTmpfs to false. Ideally, there would also be a way with nix-build to perform the build in /var/tmp instead of /tmp, which could be used when developing for builds that use large files.

Builds could also all be done in /var/tmp, but /tmp is the semantically correct location and using a tmpfs for builds makes sense given that Nix systematically removes the build folder once the build is done. Doing builds using a tmpfs should make them faster since it will download and write files to volatile memory instead of disk. Reads are already usually cached by the kernel when there is memory available.

@fpletz fpletz added this to the 17.03 milestone Nov 28, 2016
@fpletz fpletz self-assigned this Jan 20, 2017
@globin globin added this to Open in Blocking Issues 17.09 Feb 14, 2017
@fpletz fpletz modified the milestones: 17.09, 17.03 Feb 28, 2017
@fpletz
Copy link
Member

fpletz commented Feb 28, 2017

Deferred to 17.09 because a solution to use another directory on a per build basis would be needed.

@fpletz fpletz removed this from Open in Blocking Issues 17.09 Feb 28, 2017
@abbradar
Copy link
Member

Another solution may be to make Nix build in /var/tmp instead.

@lukateras lukateras removed this from the 17.09 milestone Dec 30, 2017
@ckauhaus
Copy link
Contributor

Any news on this one? Will this make it into 18.03?

@samueldr
Copy link
Member

samueldr commented Aug 2, 2018

Adopting into 18.09 not for definitive inclusion, but since it previously was part of release milestones.


I would like to know if the previous concerns are still relevant, and what work is being done to fix them, if any.


And while I was sending this, something related popped-up

[18:54:47] <symphorien> Nix is already ram hungry enough that I wouldn't like some ram to be used for /tmp
[18:55:46] <infinisil> Good point heh

This may be a valid concern, with the previously shared concerns.

@vcunat
Copy link
Member

vcunat commented Aug 3, 2018

It's a rather special use case, but I disabled this on the t4{a,b} machines on Hydra. That certainly lead to some slight slow-down, but reliability increased. The machines don't have that little RAM (2-3 GiB/thread) and "big-parallel" builds didn't go to them, but some problems still happened daily.

@samueldr samueldr modified the milestones: 18.09, 19.03 Oct 6, 2018
@infinisil
Copy link
Member

18:45 <infinisil> samueldr: I'm removing #17494 from the milestone, that alright? Nobody seems to really care much about it
18:45 <{^_^}> https://github.com/NixOS/nixpkgs/pull/17494 (by cmfwyp, 2 years ago, open): nixos: set default for boot.tmpOnTmpfs to true
18:48 <samueldr> I think a definitive action (merge / close) might need to happen with regards to the question, rather than moving it on the side
18:50 <samueldr> and with the discussion happpening in the PR, and previous discussions on different nixos irc channels, I think that it should be closed; disk space being a cheaper resource to abuse than working memory (RAM) means it probably is better to keep the current default
18:50 <samueldr> *especially* considering how not all FSes can receive a swap file
18:56 <gchristensen> +1
18:56 <gchristensen> but also kinda wish deleting temp on boot was default
18:56 <samueldr> that's probably the more right alternative to the same intents
18:57 <samueldr> though I was thinking it might need a "once you're installed" section to an installation appendix, with common configuration stories
18:57 <gchristensen> a GREAT idea
18:58 <infinisil> Alright I'll close it 

@infinisil infinisil closed this Jan 27, 2019
@infinisil infinisil removed this from the 19.03 milestone Jan 27, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet