systemd: add systemd-importd #70352

wucke13 · 2019-10-03T22:06:29Z

Motivation for this change

Adding systemd-importd to the build, so that machinectls import-.*
may actually do anything. Currently they fail with

Failed to transfer image: The name org.freedesktop.import1 was not provided by any .service files

as systemd-importd is not built.
This potentially will fix #70348

Testing

spawn a nixos vm based on this branch with gnupg and curl in the system environment and working sudo
run this (as normal user with working sudo) :

sudo gpg --no-default-keyring --keyring=/etc/systemd/import-pubring.gpg --fingerprint
tfile=$(mktemp -u /tmp/masterkey.nspawn.org.XXXXXXXXXXX)
curl https://nspawn.org/storage/masterkey.pgp -o "$tfile"
sudo gpg --no-default-keyring --keyring=/etc/systemd/import-pubring.gpg --import "$tfile"
machinectl pull-tar https://nspawn.org/storage/ubuntu/bionic/tar/image.tar.xz
machinectl start image
machinectl shell image

You will get a lot of selinux warning (which is perfectly fine if your vm has no se linux). In the end, the image should be verified by checksum and the signature proven by the imported key.

Things done

Notify maintainers

cc @

andir · 2019-10-03T23:13:20Z

I'll try to have a look at this in the next days. I vaguely remember reasons for not doing what you did there with dbus. I can't remember right now. Hopefully that gets back to me within the next days.

wucke13 · 2019-10-08T23:13:28Z

ping @andir

edolstra · 2019-10-09T10:47:18Z

What's the effect on closure size? It doesn't seem desirable to me to make systemd dependent on X.org (via libxkbdcommon).

wucke13 · 2019-10-09T12:57:28Z

What's the effect on closure size?

How would I find out?

Regarding libxkbdcommon: I have no clue why I added it, but it seems to be not necessary. I removed the unneeded deps and rebased on master. I will test 7d20b67 it in a vm and report back in a few hours.

edolstra · 2019-10-09T13:36:20Z

How would I find out?

nix path-info -S (or -rS to see the paths in the closure).

wucke13 · 2019-10-09T19:29:11Z

Ah, with the path to the nix store. I always tried nix path-info -S systemd. The closure size becomes 147.0M, so round about +35M.
While testing, I found out that there is an issue with direct calls to gpg and tar in the import code, though that should be fixed in the next push.

edolstra · 2019-10-09T20:30:50Z

Hm, I'd really prefer to avoid a hard dependency on X11 or gpg in the base system.

wucke13 · 2019-10-09T21:08:48Z

That seems reasonable. Do we have any other systemd-* component which is seperated from the systemd base expression?
X11 is not in question, that was plain wrong by me (though gpg is, for the sake of signed images).

Would you prefer a split derivation (with a var which optionally enables systemd-importd) or a more or less completely separated expression? What is it that makes being dependent on gpg bad?
Btw: b7b78a0 is fully functional:

machinectl import-tar works
machinectl pull-tar works (Inlcuding check of pgp signature and checksum
`machinectl start`` and so on works too.

So this branch may be considered a full fix to #70348 .

Btw 2:
Debian (via systemd ) and Archlinux (via pacman) for example depend on gpg for their base system. Interestingly, debian has systemd and systemd-container (so they split apart the nspawn stuff into a separate package), but gpg is a dependency of the systemd package. Obviously this might be related to their use of gpg in their chain of trust, but nonetheless having gpg in the base system doesn't seem to be terrible unpopular.

andir · 2019-10-09T22:03:39Z

Is it feasible to have those additional binaries in another output without increasing the closure of the out output?

That obviously wouldn't help against the build time dependencies. You could probably add the suggested switch and then we have another systemd package that carries all the extra bloat that most/many users do not care about.

wucke13 · 2019-10-09T23:09:35Z

Is it feasible to have those additional binaries in another output without increasing the closure of the out output?

I'm afraid that it might become difficult marrying both parts together. I will look into it though.

another systemd package that carries all the extra bloat that most/many users do not care about.

You do realize that the current state is shipping not working bloat (at least machinectl) in the normal package? Of course having finer control over what to build and whatnot is good, but the user gets bloat+regressions right now (of which at least the regressions shall go IMO). Maybe a general decision should be made on how to deal with systemd, IMHO the current package ships a lot of things which may be considered bloat since they are not at all needed for an init.

TL;DR
systemd is bloated in its nature. That may be handled by splitting packages, ... . This PR addresses a regression in nixos' systemd package but does not address systemds bloat.

Edit: I started trying to split a systemd-container package from systemd one. The package compiles but doesn't work. The old monolithic approach is is available here: https://github.com/wucke13/nixpkgs/tree/systemd-monolithic, just in case that you change your mind regarding gnupg.

What is a nice way of inheriting all the meson flags regarding folders and destinations ("-Drootlibdir=${placeholder "lib"}/lib" and alike) but not all the ones regarding features (is there something like if override )? Like, how can both be split into to distinct variables of which I can then only append to one in the systemd-container package?
When building systemd monolithic (all in one package), I had to add "systemd-importd.service" and "dbus-org.freedesktop.import1.service" to upstreamSystemUnits (in nixos/modules/system/boot/systemd.nix). How can I achieve the same for something which is not part of the systemd package?
What does meson do if I enable a feature and disable it afterwards (using the mesonFlags array)? My systemd-container derivation ends up including everything the normal systemd package provides (plus more) despite me deactivating all features but importd and machined?

Edit 2:
I have not found a way to reliably build only a a feature of systemd without the base install. It looks like most other distros (fedora) build systemd with all features and split away the results in multiple packages. Would that be a sufficient solution?

Conclusion

Building only parts of systemd (like systemd-importd without all of the base systemd-* stuff) is beyond my capabilities.
=> The way to go is either a switch in the expression or split out paths
Having eventually many switches in the systemd compile process will create a lot permutations for all the different combinations of all the different systemd packages available.
Compiling systemd with all features and stripping away the different components in many outs is beyond my capabilities too

My wish is to have machinectl working. I fixed that. Bloated systemd is another issue, which has to be addressed, but not by me. I have neither the time nor the skill to come up with a decent solution for a decomposed systemd package (and it looks like some strategic decision should be made before somebody tries to actually implement anything).

flokli · 2019-10-28T21:20:22Z

Not squashing is fine, the first is good on its own too. "systemd: etc" is what's not really saying. It could be something like: ``` systemd: add importd This will fix the `machinectl import-.*` commands which currently fail with ... ```

wucke13 · 2019-10-29T14:07:36Z

Ah I see! Done.

flokli · 2019-10-30T14:33:40Z

This doesn't seem to evaluate correctly.

wucke13 · 2019-10-31T21:17:10Z

@GrahamcOfBorg build systemd

flokli · 2019-11-02T18:55:00Z

Can you remove the gnupg-minimal = callPackage ../tools/security/gnupg/minimal.nix { }; addition / removal in between the two commits?

Basically a flavour of gnugpg, which solely containts `bin/gnupg`.

Adding `systemd-importd` to the build, so that `machinectl`s `import-.*` may actually do anything. Currently they fail with ``` Failed to transfer image: The name org.freedesktop.import1 was not provided by any .service files ``` as `systemd-importd` is not built. Also registers the regarding dbus api and service in the systemd module.

This adds a test downloading an nspawn container via http, and ensures sha256sum verification and gpg signature verification work.

flokli · 2019-11-03T13:35:25Z

I added a systemd-nspawn test which downloads an nspawn container via http from a local server, and ensures sha256sum verification and gpg signature verification work. PTAL.

flokli · 2019-11-03T13:35:35Z

@GrahamcOfBorg test systemd-nspawn

flokli · 2019-11-05T08:40:03Z

Ran tests manually on both aarch64-linux and x86_64-linux.

flokli · 2019-11-05T14:43:47Z

Added this test to the list in #72828 (it still needs to trickle from staging to master until it can be fixed there).

kirelagin · 2019-11-05T21:15:55Z

Hey, sorry for a stupid question, but how does this work? If I am reading meson.build correctly, want_importd requires zlib, but it is not in buildInputs.

wucke13 · 2019-11-05T21:48:51Z

Hey, sorry for a stupid question, but how does this work? If I am reading meson.build correctly, want_importd requires zlib, but it is not in buildInputs.

I guess things which are not explicitly deactivated are enabled on demand!

kirelagin · 2019-11-05T22:24:39Z

The thing is in this PR you set -Dimportd=true, which activates this if and then, if zlib is not found, this if should trigger and configure should fail with an error. And HAVE_ZLIB is only ever set to 1 if zlib was actually found, of course.

kirelagin · 2019-11-05T22:37:06Z

Ok, it ends up being in the dependencies closure somehow, probably as a propagated dep of some other package, huh.

systemd: add systemd-importd (cherry picked from commit c3566c7)

This reverts commit 602d5f5.

adrianparvino · 2019-11-30T12:29:02Z

Hi, this caused a regression on our machines.

58fb23f

exchange.> reloading the following units: dbus.service, firewall.service

6317f5b

exchange.> stopping the following units: dbus.service, dbus.socket, nscd.service
exchange.> Failed to create bus connection: Connection refused

Additional information: I'm using #46013 to remove polkit from our build which I believe is what resulted in the breaking combination.

flokli · 2019-11-30T12:42:32Z

I think this is the same issue as #74626 and not immediately caused by this PR.

ofborg bot requested review from andir, edolstra and lovek323 October 3, 2019 22:21

ofborg bot added 10.rebuild-darwin: 101-500 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ labels Oct 3, 2019

wucke13 force-pushed the systemd-importd branch from 854033b to f5b18e7 Compare October 3, 2019 22:28

wucke13 changed the title ~~[WIP] systemd: add systemd-importd~~ systemd: add systemd-importd Oct 3, 2019

wucke13 force-pushed the systemd-importd branch from f5b18e7 to 7d20b67 Compare October 9, 2019 12:46

wucke13 force-pushed the systemd-importd branch from 7d20b67 to 50aa916 Compare October 9, 2019 21:06

ofborg bot added 6.topic: nixos 8.has: module (update) labels Oct 9, 2019

ofborg bot requested review from fpletz, vrthra and peti October 9, 2019 21:22

wucke13 force-pushed the systemd-importd branch from 50aa916 to b7b78a0 Compare October 9, 2019 21:57

wucke13 force-pushed the systemd-importd branch 2 times, most recently from c891d8d to edf1e49 Compare October 10, 2019 11:36

ofborg bot added 8.has: package (new) 10.rebuild-darwin: 0 labels Oct 10, 2019

wucke13 changed the base branch from master to staging October 28, 2019 19:49

wucke13 force-pushed the systemd-importd branch from 2ee18b9 to f6e580c Compare October 29, 2019 14:06

FRidh moved this from WIP to Needs review in Staging Oct 29, 2019

wucke13 force-pushed the systemd-importd branch from f6e580c to b10cbda Compare October 31, 2019 15:30

wucke13 added 2 commits November 2, 2019 21:32

gnugp: adding option for minimal build

d1270ac

Basically a flavour of gnugpg, which solely containts `bin/gnupg`.

wucke13 force-pushed the systemd-importd branch from b10cbda to 29ac226 Compare November 2, 2019 20:33

nixosTests.systemd-nspawn: add test

52ed193

This adds a test downloading an nspawn container via http, and ensures sha256sum verification and gpg signature verification work.

flokli merged commit c3566c7 into NixOS:staging Nov 5, 2019

Staging automation moved this from Needs review to Done Nov 5, 2019

kirelagin mentioned this pull request Nov 5, 2019

More flexible systemd expression for minimal builds #72802

Closed

10 tasks

flokli mentioned this pull request Nov 8, 2019

Ship Systemd with systemd-importd so that machinectl is not broken #70348

Closed

dtzWill pushed a commit to dtzWill/nixpkgs that referenced this pull request Nov 11, 2019

Merge pull request NixOS#70352 from wucke13/systemd-importd

602d5f5

systemd: add systemd-importd (cherry picked from commit c3566c7)

dtzWill added a commit to dtzWill/nixpkgs that referenced this pull request Nov 11, 2019

Revert "Merge pull request NixOS#70352 from wucke13/systemd-importd"

d4167a7

This reverts commit 602d5f5.

flokli mentioned this pull request Apr 30, 2020

machinectl does not work #12347

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

systemd: add systemd-importd #70352

systemd: add systemd-importd #70352

wucke13 commented Oct 3, 2019 •

edited

andir commented Oct 3, 2019 •

edited

wucke13 commented Oct 8, 2019 •

edited

edolstra commented Oct 9, 2019

wucke13 commented Oct 9, 2019

edolstra commented Oct 9, 2019

wucke13 commented Oct 9, 2019

edolstra commented Oct 9, 2019

wucke13 commented Oct 9, 2019 •

edited

andir commented Oct 9, 2019 •

edited

wucke13 commented Oct 9, 2019 •

edited

flokli commented Oct 28, 2019 via email

wucke13 commented Oct 29, 2019

flokli commented Oct 30, 2019

wucke13 commented Oct 31, 2019

flokli commented Nov 2, 2019

flokli commented Nov 3, 2019

flokli commented Nov 3, 2019

flokli commented Nov 5, 2019

flokli commented Nov 5, 2019

kirelagin commented Nov 5, 2019

wucke13 commented Nov 5, 2019

kirelagin commented Nov 5, 2019 •

edited

kirelagin commented Nov 5, 2019 •

edited

adrianparvino commented Nov 30, 2019 •

edited

flokli commented Nov 30, 2019 via email

Navigation Menu

systemd: add systemd-importd #70352

systemd: add systemd-importd #70352

Conversation

wucke13 commented Oct 3, 2019 • edited

Motivation for this change

Testing

Things done

Notify maintainers

andir commented Oct 3, 2019 • edited

wucke13 commented Oct 8, 2019 • edited

edolstra commented Oct 9, 2019

wucke13 commented Oct 9, 2019

edolstra commented Oct 9, 2019

wucke13 commented Oct 9, 2019

edolstra commented Oct 9, 2019

wucke13 commented Oct 9, 2019 • edited

andir commented Oct 9, 2019 • edited

wucke13 commented Oct 9, 2019 • edited

Conclusion

flokli commented Oct 28, 2019 via email

wucke13 commented Oct 29, 2019

flokli commented Oct 30, 2019

wucke13 commented Oct 31, 2019

flokli commented Nov 2, 2019

flokli commented Nov 3, 2019

flokli commented Nov 3, 2019

flokli commented Nov 5, 2019

flokli commented Nov 5, 2019

kirelagin commented Nov 5, 2019

wucke13 commented Nov 5, 2019

kirelagin commented Nov 5, 2019 • edited

kirelagin commented Nov 5, 2019 • edited

adrianparvino commented Nov 30, 2019 • edited

flokli commented Nov 30, 2019 via email

wucke13 commented Oct 3, 2019 •

edited

andir commented Oct 3, 2019 •

edited

wucke13 commented Oct 8, 2019 •

edited

wucke13 commented Oct 9, 2019 •

edited

andir commented Oct 9, 2019 •

edited

wucke13 commented Oct 9, 2019 •

edited

kirelagin commented Nov 5, 2019 •

edited

kirelagin commented Nov 5, 2019 •

edited

adrianparvino commented Nov 30, 2019 •

edited