[WIP] ipfs-cluster: systemd service #100871

lordcirth · 2020-10-17T20:13:42Z

Making this WIP PR to avoid someone duplicating effort while I work on it. It will need to be squashed.

Motivation for this change

ipfs-cluster is meant to be run as a daemon

Things done

Created systemd service for ipfs-cluster, and an -init service, with some basic settings

aanderse

Thanks for writing this service. ipfs seems to be a hot topic lately, so I'm sure this service will be appreciated 👍

I hope you find my suggestion helpful. Please don't hesitate to ask if anything I have written isn't clear. I'm addition to my comments I have a few questions:

Do you know if upstream provides a systemd unit at all?
Have you considered any systemd hardening on this service?
Are any firewall holes required for proper operation?
Does this program take a configuration file at all?

nixos/modules/services/network-filesystems/ipfs-cluster.nix

aanderse · 2020-10-18T00:16:29Z

nixos/modules/services/network-filesystems/ipfs-cluster.nix

+
+    systemd.packages = [ pkgs.ipfs-cluster pkgs.coreutils ];
+
+    systemd.services.ipfs-cluster-init = {


Could this entire service could be replaced with an ExecStartPre?

I don't think so? I am using unitConfig.ConditionDirectoryNotEmpty = "!${cfg.dataDir}"; on -init to prevent it being run every time.

It seems like you could have a preStart script that just checks if the directory exists then calls the init command if not. Save the hassle of a second unit. That's sorta what ExecStartPre is for.

nixos/modules/services/network-filesystems/ipfs-cluster.nix

aanderse · 2020-10-18T00:26:44Z

nixos/modules/services/network-filesystems/ipfs-cluster.nix

+  opt = options.services.ipfs-cluster;
+
+  # secret is by envvar, not flag
+  initFlags = toString [


Maybe initFlags = optionalString (cfg.initPeers != []) "--peers ${lib.concatStringsSep "," cfg.initPeers}";?

I figured there would be more flags soon enough

lordcirth · 2020-10-18T02:50:21Z

Thanks for writing this service. ipfs seems to be a hot topic lately, so I'm sure this service will be appreciated +1

I hope you find my suggestion helpful. Please don't hesitate to ask if anything I have written isn't clear. I'm addition to my comments I have a few questions:
* Do you know if upstream provides a `systemd` unit at all?

* Have you considered any `systemd` hardening on this service?

* Are any firewall holes required for proper operation?

* Does this program take a configuration file at all?

Thank you for your feedback!

Upstream does not provide a .service file
I have not yet done any hardening, besides ensuring it runs as "ipfs". What kind of hardening would you consider?
It will require firewall exceptions to be added, yes
"ipfs-cluster-service init" creates $IPFS_CLUSTER_PATH/service.json automatically; the only mandatory option is "--consensus", with --peers and the secret covering the common uses

Thanks for pointing out the problem with readFile; git grepping nixpkgs, I see that duplicity uses EnvironmentFile = cfg.secretFile to have systemd read the secret at runtime.

AluisioASG · 2020-10-19T21:28:57Z

This will need a way to change arbitrary settings (in particular, all the addresses in the config). I wrote a module a while ago that uses the environment variable overrides to enable that while avoiding edits to service.json. On second thought you might be able to merge settings into it with something like jq's recursive merge operator as well.

lordcirth · 2020-12-11T19:08:39Z

Rebased to current master.

bqv · 2021-01-08T03:44:38Z

Very cool, +1 interest

mmahut · 2021-02-12T14:09:44Z

Any updates on this PR, please?

lordcirth · 2021-02-16T21:53:36Z

Any updates on this PR, please?

Yeah, sorry. I'm going to rebase and try to test that minimal functionality is working.

mmahut · 2021-02-17T13:54:13Z

Would there be a way to write a test for this service?

nixos/modules/services/network-filesystems/ipfs-cluster.nix

lordcirth · 2021-02-17T15:44:24Z

Would there be a way to write a test for this service?

You'd need to spawn 2 containers, connect them, pass data back and forth. I think this PR should go ahead without automated tests. It's been taking long enough. I'm trying to figure out nixos-build-vms to do some tests.

lordcirth · 2021-02-18T03:26:33Z

Ok, tested the following so far:

init without secret generates one
init with secretFile works
initPeers works (only on first run)
daemon comes up on reboot
crdt mode works

nixos/modules/services/network-filesystems/ipfs-cluster.nix

mmahut · 2021-03-11T16:22:30Z

Should we manage the user for the service?

Starting ipfs-cluster-init.service...
ipfs-cluster-init.service: Failed to determine user credentials: No such process
ipfs-cluster-init.service: Failed at step USER spawning /nix/store/yvdiqsnw2d5f04wcc6lgz5hl30711dfz-ipfs-cluster-0.13.0/bin/ipfs-cluster-service: No such process

bachp · 2021-03-17T21:24:15Z

nixos/modules/services/network-filesystems/ipfs-cluster.nix

+        Group = cfg.group;
+      };
+    };
+    networking.firewall.allowedTCPPorts = [ 9096 ];


I don't think just opening a firewall port here is a good idea.

If you want this it should be hidden behind a openFirewall option that is off by default.

It's a peer-to-peer service; it's expected that peers can connect. If enough other peers are available in the configured cluster, it should work without an open port, but not as well. In many configurations it just won't do anything without an open port.

@lordcirth I think it's still best practice in nixpkgs to make it explicit via an openFirewall option

lordcirth · 2021-03-18T23:27:04Z

Should we manage the user for the service?

Starting ipfs-cluster-init.service...
ipfs-cluster-init.service: Failed to determine user credentials: No such process
ipfs-cluster-init.service: Failed at step USER spawning /nix/store/yvdiqsnw2d5f04wcc6lgz5hl30711dfz-ipfs-cluster-0.13.0/bin/ipfs-cluster-service: No such process

It uses the ipfs user by default, which is created by the ipfs service. ipfs-cluster is largely useless without an ipfs daemon anyway.

aanderse · 2021-03-18T23:48:16Z

@lordcirth I present "exhibit A": security.acme.acceptTerms... which does absolutely nothing at all.

Sometimes we just need the user to acknowledge that a module is implicitly doing something, by explicitly stating such 🤷‍♂️

mmahut · 2021-03-29T10:38:57Z

What is the different between initPeers and having a bootstrap peers?

fricklerhandwerk · 2021-05-15T14:23:04Z

nixos/modules/services/network-filesystems/ipfs-cluster.nix

+    services.ipfs-cluster = {
+
+      enable = mkEnableOption
+        "Pinset orchestration for IPFS - requires ipfs daemon to be useful";


This reads like I'd have to do something with ipfs daemon to make this work, but the daemon is started by this service, right?

While the first paragraph from their repo is not all that much more informative, it at least contains more information to understand what this is even about:

IPFS Cluster provides data orchestration across a swarm of IPFS daemons by allocating, replicating and tracking a global pinset distributed among multiple peers.

No, this starts the ipfs-cluster daemon. You would usually want to enable the ipfs daemon as well, but it's not a hard dependency as you could be running ipfs via a different module, in a separate container, etc.

No default so the user can't miss it.

fricklerhandwerk · 2021-05-15T14:49:53Z

nixos/modules/services/network-filesystems/ipfs-cluster.nix

@@ -82,6 +82,7 @@ in {
      wantedBy = [ "default.target" ];

      serviceConfig = {
+        # "" clears exec list (man systemd.service -> execStart)


This is such an obscure thing to do, seriously systemd... I had to read carefully to find that, and it seems it does not even apply to all list-type options.
https://www.freedesktop.org/software/systemd/man/systemd.service.html#ExecStart=

fricklerhandwerk · 2021-05-15T14:50:10Z

LGTM

srid · 2021-09-03T19:54:53Z

Is there anything preventing us from getting this merged?

(I'm looking forward to using it, or even testing it!)

mohe2015 · 2021-09-03T19:57:00Z

Is there anything preventing us from getting this merged?

(I'm looking forward to using it, or even testing it!)

You should test this before this get's merged so we know it works :D (at least that would be great)

stale · 2022-04-18T17:16:11Z

I marked this as stale due to inactivity. → More info

McSinyx · 2022-05-02T15:14:57Z

1.0.0 has recently been out with breaking changes, need anything be done aside from the version bump here?

lordcirth · 2022-05-05T18:13:42Z

1.0.0 has recently been out with breaking changes, need anything be done aside from the version bump here?

"There are no breaking configuration changes". Sounds like the breaking is in the API.

1000101 · 2022-06-11T08:56:56Z

👀

1000101 · 2022-07-21T12:53:09Z

Anything that's blocking the merge? I've been running servers based on this PR for more than a year now.

McSinyx · 2022-09-29T19:28:43Z

@lordcirth, could you please rebase this on top of master or unstable?

bachp · 2022-10-29T21:01:02Z

@lordcirth Are you still working on this MR?

lordcirth · 2022-11-23T18:39:17Z

@lordcirth Are you still working on this MR?

Not really, sorry.

ofborg bot added 6.topic: nixos 8.has: module (update) 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 10.rebuild-linux: 1 labels Oct 17, 2020

aanderse requested changes Oct 18, 2020

View reviewed changes

lordcirth force-pushed the ipfs-cluster-service branch from e4fdac8 to 5747989 Compare December 11, 2020 17:26

lordcirth force-pushed the ipfs-cluster-service branch from 5747989 to aa7dc2e Compare January 8, 2021 06:14

bqv mentioned this pull request Jan 9, 2021

ipfs-cluster: 0.13.0 -> unstable-2020-10 #108822

Merged

10 tasks

lordcirth force-pushed the ipfs-cluster-service branch from aa7dc2e to 70f514b Compare February 17, 2021 00:04

mmahut reviewed Feb 17, 2021

View reviewed changes

nixos/modules/services/network-filesystems/ipfs-cluster.nix Show resolved Hide resolved

srid reviewed Mar 1, 2021

View reviewed changes

nixos/modules/services/network-filesystems/ipfs-cluster.nix Outdated Show resolved Hide resolved

bachp requested changes Mar 17, 2021

View reviewed changes

lordcirth added 4 commits May 14, 2021 10:34

ipfs-cluster: start service file by copying from ipfs

e31cfa7

ipfs-cluster: fix braces and add to module list

c91b1d5

ipfs-cluster: fix leftover ipfs bits

b36459e

ipfs-cluster: enable service

705a34b

lordcirth added 8 commits May 14, 2021 10:34

ipfs-cluster: re-add secret

7e2a00f

ipfs-cluster: reviewer improvements

7e40640

ipfs-cluster: fix secret handling

6fd58c4

ipfs-cluster: fix cluster var name and fix paths

9e0d1e7

ipfs-cluster: remove null from consensus

84743e3

ipfs-cluster: open TCP 9096

817dd94

ipfs-cluster: add secret generation to doc

22e2eff

ipfs-cluster: move secretFile to init

adc9727

lordcirth force-pushed the ipfs-cluster-service branch from 70f514b to adc9727 Compare May 14, 2021 14:35

fricklerhandwerk reviewed May 15, 2021

View reviewed changes

lordcirth added 2 commits May 15, 2021 10:34

Improve docs as per feedback

f3ade98

ipfs-cluster: Make openSwarmPort configurable

97fdc3d

No default so the user can't miss it.

fricklerhandwerk reviewed May 15, 2021

View reviewed changes

stale bot added 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md and removed 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md labels Apr 18, 2022

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 18, 2023

sorki mentioned this pull request Feb 29, 2024

nixos/ipfs-cluster: init #292290

Open

13 tasks

wegank added the 2.status: merge conflict label Mar 20, 2024

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Mar 20, 2024


		systemd.packages = [ pkgs.ipfs-cluster pkgs.coreutils ];

		systemd.services.ipfs-cluster-init = {

[WIP] ipfs-cluster: systemd service #100871

Are you sure you want to change the base?

[WIP] ipfs-cluster: systemd service #100871

Conversation

lordcirth commented Oct 17, 2020 • edited

Motivation for this change

Things done

aanderse left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lordcirth commented Oct 18, 2020

AluisioASG commented Oct 19, 2020

lordcirth commented Dec 11, 2020

bqv commented Jan 8, 2021

mmahut commented Feb 12, 2021

lordcirth commented Feb 16, 2021

mmahut commented Feb 17, 2021

lordcirth commented Feb 17, 2021

lordcirth commented Feb 18, 2021

mmahut commented Mar 11, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lordcirth commented Mar 18, 2021

aanderse commented Mar 18, 2021

mmahut commented Mar 29, 2021

fricklerhandwerk May 15, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fricklerhandwerk May 15, 2021 • edited

Choose a reason for hiding this comment

fricklerhandwerk commented May 15, 2021

srid commented Sep 3, 2021

mohe2015 commented Sep 3, 2021 • edited

stale bot commented Apr 18, 2022

McSinyx commented May 2, 2022

lordcirth commented May 5, 2022

1000101 commented Jun 11, 2022

1000101 commented Jul 21, 2022

McSinyx commented Sep 29, 2022

bachp commented Oct 29, 2022

lordcirth commented Nov 23, 2022

lordcirth commented Oct 17, 2020 •

edited

fricklerhandwerk May 15, 2021 •

edited

fricklerhandwerk May 15, 2021 •

edited

mohe2015 commented Sep 3, 2021 •

edited