Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reworked Cassandra Module #31410

Merged
merged 1 commit into from Aug 7, 2018
Merged

Reworked Cassandra Module #31410

merged 1 commit into from Aug 7, 2018

Conversation

fuzzy-id
Copy link
Contributor

@fuzzy-id fuzzy-id commented Nov 8, 2017

Motivation for this change

This is a rewrite of the cassandra module as described in #31349.

Things done

Cassandra succesfully starts in a container with the following expression:

  containers.cassandra =
    { privateNetwork = true;
      config =
        { config, pkgs, callPackage, ... }:
        {
          imports =
            [ /path/to/nixpkgs/nixos/modules/services/databases/cassandra.nix
            ];

          nixpkgs.config.allowUnfree = true;
           
          ids.uids.cassandra = 284;
          ids.gids.cassandra = 284;

          services.cassandra.enable = true;
          services.cassandra.package =
            pkgs.callPackage /path/to/nixpkgs/pkgs/servers/nosql/cassandra/3.11.nix {
              jre = pkgs.jre8;
            };
          services.cassandra.extraConfig =
           { commitlog_sync_batch_window_in_ms = 3;
           };
        };
    };

The module surely needs some further refinements. But it's a start. :)

@NeQuissimus
Copy link
Member

Could you add back at least a simple test?

@roberth
Copy link
Member

roberth commented Nov 15, 2017

I have tried to port the old test, but I ran into some problems that I believe should be fixed out-of-the-box.

  • JMX was not enabled, because of missing cassandra-env.sh.
  • The Thrift interface was not enabled
  • cqlsh, nodetool are not on the system path.

Other things I've noticed:

  • Other config files are also missing in cassandraEtc. Makes me wonder what else will break.
  • The repair service+timer is missing. Regular repairs seem to be a necessity, but auto-scheduled repairs have not been integrated into cassandra itself yet.

@NeQuissimus
Copy link
Member

Yeah, it's fine if there is stuff that does not work. But I'd rather we have a simplistic test than none at all...

@disassembler
Copy link
Member

based on @roberth testing and review, marking as a WIP so it doesn't get accidentally merged before those points are addressed. I also think some of these commits can be squashed. I don't see a reason to do 2 version bumps in a single PR.

@cransom
Copy link
Contributor

cransom commented Nov 15, 2017

Agreed with @NeQuissimus , the previous commit/test may not have been pretty, but it did reveal bugs in the packaging before that yielded unusable nixos packages. cassandra would run, but if you tried to join it to a cluster, it would segfault due to a missing compression library.

scratch that about the repair. it did exist.

@fuzzy-id
Copy link
Contributor Author

Couldn't find the time to push this any further earlier. I just pushed a simplistic test which already revealed a few problems. Do I have to plug in this test anywhere else?

@roberth cassandra-env.sh from the original package is now symlinked into cassandraEtc. Should we add a configuration option to overwrite this?

Should cqlsh and nodetool really go into PATH automatically? You can simply achieve the same by adding the package to environment.systemPackages. Is there some kind of policy on how these things are approached?

@disassembler I'm willing to clean up the commit history before this get merged. Should I do this in this branch and do a forced push or open a new pull request?

@fuzzy-id fuzzy-id force-pushed the fix-cassandra branch 2 times, most recently from bb2b2e4 to 47e81a5 Compare December 6, 2017 10:58
@fuzzy-id
Copy link
Contributor Author

fuzzy-id commented Dec 6, 2017

Just cleaned up the commit history.

@fuzzy-id fuzzy-id force-pushed the fix-cassandra branch 2 times, most recently from 8fb808f to 0c9c895 Compare December 13, 2017 13:18
@fuzzy-id fuzzy-id force-pushed the fix-cassandra branch 3 times, most recently from 9e7e76f to 394ead3 Compare January 3, 2018 19:33
@fuzzy-id
Copy link
Contributor Author

fuzzy-id commented Jan 3, 2018

Hey all,

I just added a two-node setup for tests. In the process of this I added the options listenAddress, listenInterface, rpcAddress and rpcInterface.

The tests run successfully here. Is there any further missing bit I oversee which would block this module from being merged?

@roberth
Copy link
Member

roberth commented Jan 4, 2018

Did you try restoring the node failure tests? They look quite simple.

@fuzzy-id fuzzy-id force-pushed the fix-cassandra branch 2 times, most recently from e431061 to 3a99935 Compare January 5, 2018 12:34
@fuzzy-id
Copy link
Contributor Author

fuzzy-id commented Jan 5, 2018

@roberth I succesfully ported the break and fix node test.

However, the replace crashed node doesn't work. Anyways, reading https://cassandra.apache.org/doc/latest/operating/topo_changes.html it is not clear at all to me how the original test could pass on Cassandra 3.11. For now, the test is commented out.

EDIT: My bad, I didn't see the extra replacement parameters passed to the node.

@fuzzy-id
Copy link
Contributor Author

fuzzy-id commented Jan 5, 2018

Just fixed the last test. All of them are passing!

@fuzzy-id
Copy link
Contributor Author

fuzzy-id commented Jan 8, 2018

Is there anything else blocking this from being merged?

@roberth
Copy link
Member

roberth commented Jan 8, 2018

You can split out the version updates like disassembler requested. Otherwise, looks good to me.

@fuzzy-id
Copy link
Contributor Author

fuzzy-id commented Jan 8, 2018

@roberth What do you mean? Should I drop e76e9eaf91c all together?

@roberth
Copy link
Member

roberth commented Jan 8, 2018

Well, you asked what else could be done ;). Separate PRs may help to get some of the work merged more quickly. It also fits the contributing guidelines a bit better, although that is up for interpretation.

@fuzzy-id
Copy link
Contributor Author

fuzzy-id commented Jan 9, 2018

So I tried to separate concerns and bump the versions of all cassandra packages currently in nixpkgs in a separate branch. As I didn't want to go through testing them manually I reconfigured the tests of this pull request. They all fail except for 3.1!

So how should we deal with this? Drop all packages except 3.1? Note that 2.1, 2.2 and 3.0 are still receiving bug fixes upstream.

Should the module be changed so it can only be used with Cassandra 3.1?

I don't have the time and energy to make the service-module cross-version compatible!

@roberth
Copy link
Member

roberth commented Feb 8, 2018

@fuzzy-id The lines from your example

      ids.uids.cassandra = 284;
      ids.gids.cassandra = 284;

seem to be required, whereas for other services these ids are added automatically.

@fuzzy-id
Copy link
Contributor Author

fuzzy-id commented Feb 8, 2018

@roberth I added a small section to the release notes.

Previously the uid and gid were set in the service itself. Should I put it back in there? Does this also work around the possibility of overlapping ids?

EDIT: Just to clarify this: I added a cassandra user and group to nixos/modules/misc/ids.nix

@roberth
Copy link
Member

roberth commented Feb 9, 2018

@fuzzy-id I'm sorry, my last comment about the ids was incorrect. I jumped to a conclusion too quickly. Your solution seems to be an improvement. I believe NixOS has a 'reserved' range of uids for this purpose, where you can consider the uid assigned when it gets merged. Haven't seen official docs about that though. LGTM

@roberth
Copy link
Member

roberth commented Feb 9, 2018

@NeQuissimus do you think this is ready for merging?

@fuzzy-id
Copy link
Contributor Author

removed release note and resolved conflict with master branch.

@fuzzy-id fuzzy-id mentioned this pull request Apr 27, 2018
8 tasks
@fuzzy-id fuzzy-id changed the title Fix cassandra Reworked Cassandra Module Apr 27, 2018
@roberth
Copy link
Member

roberth commented Apr 27, 2018

@fuzzy-id This does need to be in the release notes. Would you mind moving it to the 18.09 release notes?

@teburd teburd mentioned this pull request May 8, 2018
@teburd
Copy link
Contributor

teburd commented May 8, 2018

I would really like to see this working again

@fuzzy-id
Copy link
Contributor Author

fuzzy-id commented May 9, 2018

@BFrog This is working! We don't use it in production atm, as we are in alpha phase. Hence, I haven't played around and cleaned up cluster-setups. But for simple single-machine setups it works like a charm. (NixOS-tests for multi-cluster setups are there, so it should be working.)

@fuzzy-id
Copy link
Contributor Author

fuzzy-id commented Aug 6, 2018

Hey guys,

is anyone willing to merge this, so that it will be part of 18.09? I could invest some more cleanup work if there is the need to. (i.e. adding the release note and resolve the conflicts). But it would be nice to get some affirmation that the work is not going to be lost because no one really has the time to merge this.

I just wanted to do version bumps for Cassandra and all this rebasing and cherry-picking just to be able to run the tests is a big pain in the as. So I guess we would rather maintain our own versions of this module internally.

@NeQuissimus
Copy link
Member

If you rebase it, we should be able to merge it.

Adds a replacement for the previously broken
`services.database.cassandra` with tests for a multi-node setup.
@fuzzy-id
Copy link
Contributor Author

fuzzy-id commented Aug 7, 2018

Rebased and added release note. Let me know if there is something missing. I also re-ran the tests for all Cassandra versions.

@NeQuissimus NeQuissimus merged commit 31e11bd into NixOS:master Aug 7, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants