Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nixos/bitcoind: change to multi-instance + add tests #93700

Merged
merged 4 commits into from Jul 27, 2020
Merged

nixos/bitcoind: change to multi-instance + add tests #93700

merged 4 commits into from Jul 27, 2020

Conversation

1000101
Copy link
Member

@1000101 1000101 commented Jul 23, 2020

Motivation for this change

Currently, it is not possible to run multiple instances of bitcoind (typical use case is running mainnet and testnet on the same server). While at it, I also refactored some things which I think weren't properly configured (e.g. RPC options were ignored when running testnet) or deprecated (loaOf). Plus I have also added basic tests.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@mmahut
Copy link
Member

mmahut commented Jul 24, 2020

Please add this change to the release notes for the next release.

nixos/modules/services/networking/bitcoind.nix Outdated Show resolved Hide resolved
nixos/modules/services/networking/bitcoind.nix Outdated Show resolved Hide resolved
@aanderse
Copy link
Member

This dramatically increases the complexity of this module. Who is the maintainer? Are containers not an adequate way to run multiple instances of this service on the same machine?

@mmahut @prusnak is the added complexity "worth it"?

@mmahut
Copy link
Member

mmahut commented Jul 26, 2020

@aanderse hmm, good point, but I don't consider this to be a major rewrite, at least running both testnet and mainnet networks at the same time is a standard feature that has been omitted in the previous module. I like the rewrite, the previous code was a mess (understandable, if we consider what a mess the upstream bitcoin node is).

@1000101 as mentioned, please note this change in the release notes and if possible, add yourself as maintainer (I guess you are fine with it given you added yourself to the test?).

@aanderse
Copy link
Member

@mmahut thanks for feedback.

@1000101
Copy link
Member Author

1000101 commented Jul 27, 2020

This dramatically increases the complexity of this module. Who is the maintainer? Are containers not an adequate way to run multiple instances of this service on the same machine?

@mmahut @prusnak is the added complexity "worth it"?

@aanderse It may look like it, but it's just a rewrite. In fact, everything will work out of box (as previously), all you have to do is name the instance and that's it. I've updated some of stuff I found not working properly but other than that, I kept it compatible with the original service (apart from the instance name).

There was no maintainer before, but it's a good point, I'll add myself.

@1000101
Copy link
Member Author

1000101 commented Jul 27, 2020

@aanderse hmm, good point, but I don't consider this to be a major rewrite, at least running both testnet and mainnet networks at the same time is a standard feature that has been omitted in the previous module. I like the rewrite, the previous code was a mess (understandable, if we consider what a mess the upstream bitcoin node is).

@1000101 as mentioned, please note this change in the release notes and if possible, add yourself as maintainer (I guess you are fine with it given you added yourself to the test?).

@mmahut yeap, will do

@1000101
Copy link
Member Author

1000101 commented Jul 27, 2020

@GrahamcOfBorg test bitcoind

@mmahut
Copy link
Member

mmahut commented Jul 27, 2020

Looks good, thank you.

@mmahut mmahut merged commit b415eba into NixOS:master Jul 27, 2020
@emmanuelrosa
Copy link
Contributor

This changes the bitcoind data directory, correct? That means that if there's already a running instance which is using /var/lib/bitcoind/ as the data directory, this change would ignore that and bitcoind would start doing an initial sync.

There's also the issue of changing the bitcoind user, since existing files would be owned by the original bitcoind user.

You may need to use NixOS stateVersion (and accompaning release notes) to prevent this change from applying automatically to users which already run bitcoind.

@mmahut
Copy link
Member

mmahut commented Jul 28, 2020

Given this is breaking the way this service is configured I think just mentioning it in the release notes is enough, given this change will not redefine the directory nor user out of the box without users intervention.

But you have a good point that it would be useful if the release notes did mention this change, just in case. @1000101 do you mind opening an addendum?

@emmanuelrosa
Copy link
Contributor

To use this new multi-instance config with an existing bitcoind data directory, the user would have to change from something like this:

  services.bitcoind = {
    enable = true;
    extraConfig = "...";
    ...
  };

To something like this:

  services.bitcoind.mainnet = {
    enable = true;
    dataDir = "/var/lib/bitcoind";
    user = "bitcoin";
    extraConfig = "...";
    ...
  };

The key settings are:

  • dataDir - to continue using the same data directory
  • user - to continue using the same user so that bitcoind maintains access to its files.

@1000101
Copy link
Member Author

1000101 commented Jul 29, 2020

This changes the bitcoind data directory, correct? That means that if there's already a running instance which is using /var/lib/bitcoind/ as the data directory, this change would ignore that and bitcoind would start doing an initial sync.

There's also the issue of changing the bitcoind user, since existing files would be owned by the original bitcoind user.

The directory and user may/will change by default, but you have the option to set them manually to the original (desired) values. However, you are absolutely right and I should mention this in the release notes (incompatibility section).

You may need to use NixOS stateVersion (and accompaning release notes) to prevent this change from applying automatically to users which already run bitcoind.

Since it is a breaking change it will not automatically change, however, I definitely need to add the points you mention in the release notes.

@1000101
Copy link
Member Author

1000101 commented Jul 29, 2020

To use this new multi-instance config with an existing bitcoind data directory, the user would have to change from something like this:

  services.bitcoind = {
    enable = true;
    extraConfig = "...";
    ...
  };

To something like this:

  services.bitcoind.mainnet = {
    enable = true;
    dataDir = "/var/lib/bitcoind";
    user = "bitcoin";
    extraConfig = "...";
    ...
  };

The key settings are:

* `dataDir` - to continue using the same data directory

* `user` - to continue using the same user so that bitcoind maintains access to its files.

Thank you so much, I'll amend it.

Copy link
Member

@aanderse aanderse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity would it make sense to use DynamicUser at all here?

nixos/doc/manual/release-notes/rl-2009.xml Show resolved Hide resolved
@emmanuelrosa
Copy link
Contributor

If DynamicUser changes the user ID then I don't recommend using it (for users already using bitcoind) because then bitcoind may not be able to access the files in it's data directory.

@aanderse
Copy link
Member

aanderse commented Aug 1, 2020

DynamicUser keeps all data under StateDirectory owned by the user the service runs by, even when that user changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants