scylladb: init at 3.0.5 #61438

workflow · 2019-05-13T07:55:33Z

Motivation for this change

Initial build of ScyllaDB, the real-time big data database.

This is an MVP, let me know what should be cleaned up.

Warning: Long build ahead!

cc: @dingxiangfei2009 @snawaz

Things done

pkgs/servers/scylladb/default.nix

pkgs/development/libraries/libsystemtap/default.nix

pkgs/development/libraries/libantlr3cpp/default.nix

maintainers/maintainer-list.nix

pkgs/servers/scylladb/seastar-configure-script-paths.patch

pkgs/servers/scylladb/default.nix

workflow · 2019-05-19T12:53:14Z

@infinisil Addressed all comments. Thank you very much for your review!

nixos-discourse · 2019-07-17T08:54:06Z

This pull request has been mentioned on Nix community. There might be relevant details there:

https://discourse.nixos.org/t/prs-already-reviewed/2617/34

aanderse

I've listed a few changes I hope you'll find helpful.

pkgs/development/libraries/libantlr3cpp/default.nix

aanderse · 2019-07-17T10:09:46Z

pkgs/development/libraries/libantlr3cpp/default.nix

+  '';
+
+  meta = with stdenv.lib; {
+    description = "C++ runtime libraries of ANTLR v3";


Given the installPhase you have provided this sounds misleading. What is the difference between this package and antlr?

Good question! For some reason, even though the antlr package provides libraries, these are currently at version 2.7.7 which is incompatible with 3.x, and the existing antlr3 package does not include any libraries.

Maybe I'm missing something though?

Oh I see... If I had to guess and if I trusted my memory on the issue (which I definitely do not 😄) I would wonder if whenantlr3 was packaged for nixpkgs back in 2016 maybe the c++ support in antlr3 wasn't that good so it was just skipped... 🤷‍♂️

Maybe we should just amend the existing package?

Okay good point, I'll look into that!

Amended the existing package instead! Let me know if that looks okay, please.

pkgs/development/libraries/libantlr3cpp/default.nix

pkgs/development/libraries/libsystemtap/default.nix

pkgs/servers/scylladb/default.nix

aanderse · 2019-07-17T10:22:31Z

pkgs/servers/scylladb/default.nix

+
+  configurePhase = ''
+     patchShebangs ./configure.py
+    ./configure.py --mode=release


How about using configureFlags instead.

Great point!

What I'm trying to do, instead of overriding the configurePhase, is this:

configureScript = "./configure.py"; configureFlags = [ "--mode=release" ]; dontAddPrefix = true;

However for some reason, whenever cmake is available as a nativeBuildInput, it ignores my overrides and tries to use cmake instead.

To be more precise, it adds a fixing cmake files... phase, which I can't seem to turn off and somehow leads to this:

configuring fixing cmake files... cmake flags: -GNinja -DCMAKE_FIND_PACKAGE_NO_SYSTEM_PACKAGE_REGISTRY=ON -DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON -DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_SKIP_BUILD_RPATH=ON -DBUILD_TESTING=OFF -DCMAKE_INSTALL_INCLUDEDIR=/nix/store/gccy6z2y06s0k8z k743rar66438v0y7q-scylladb-3.0.5/include -DCMAKE_INSTALL_LIBDIR=/nix/store/gccy6z2y06s0k8zk743rar66438v0y7q-scylladb-3.0.5/lib -DCMAKE_INSTALL_NAME_DIR=/nix/store/gccy6z2y06s0k8zk743rar66438v0y7q-scylladb-3.0.5/lib -DCMAKE_POLICY_DEFAULT_CMP0025=NEW -DCMAKE_OSX_DEPLOYMENT_TARGET= - DCMAKE_OSX_SYSROOT= -DCMAKE_FIND_FRAMEWORK=last -DCMAKE_STRIP=/nix/store/6k95q6vvnmspf3c0nm8q55sixsvmn9nc-binutils-2.31.1/bin/strip -DCMAKE_RANLIB=/nix/store/6k95q6vvnmspf3c0nm8q55sixsvmn9nc-binutils-2.31.1/bin/ranlib -DCMAKE_AR=/nix/store/6k95q6vvnmspf3c0nm8q55sixsvmn9nc-binutils- 2.31.1/bin/ar -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -- The C compiler identification is GNU 8.3.0 -- The CXX compiler identification is GNU 8.3.0 -- Check for working C compiler: /nix/store/lfhwv8kc8c94ix3y7bn34zz4k1c4zgha-gcc-wrapper-8.3.0/bin/gcc -- Check for working C compiler: /nix/store/lfhwv8kc8c94ix3y7bn34zz4k1c4zgha-gcc-wrapper-8.3.0/bin/gcc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /nix/store/lfhwv8kc8c94ix3y7bn34zz4k1c4zgha-gcc-wrapper-8.3.0/bin/g++ -- Check for working CXX compiler: /nix/store/lfhwv8kc8c94ix3y7bn34zz4k1c4zgha-gcc-wrapper-8.3.0/bin/g++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done CMake Error at CMakeLists.txt:9 (message): This CMakeLists.txt file is only valid for use in IDEs, please define FOR_IDE to acknowledge this. -- Configuring incomplete, errors occurred!

pkgs/servers/scylladb/default.nix

pkgs/development/libraries/libsystemtap/default.nix

FRidh · 2019-07-17T12:17:10Z

pkgs/development/libraries/thrift/0.10.nix

+  #enableParallelBuilding = true; problems on hydra
+
+  # Workaround to make the python wrapper not drop this package:
+  # pythonFull.buildEnv.override { extraLibs = [ thrift ]; }


That's not how it should be done. Apparently this provides Python bindings. Those bindings should be made available via python-packages.nix.

Thanks for pointing that out!

Would you be okay if I look at that in a separate PR?

All I did here with thrift here is to reactivate the existing 0.10 thrift from nixpkgs, it's the vanilla thrift 0.10 so to speak.

pkgs/servers/scylladb/default.nix

workflow · 2019-07-24T02:28:06Z

@aanderse Thanks for your wonderfully detailed review! Learned a lot, and addressed all comments I believe.

workflow · 2019-07-25T03:40:39Z

@FRidh Thanks for your review as well! Please see my comments.

What are the next steps to get this merged?

aanderse · 2019-08-02T00:41:26Z

@workflow Great work! How about you squash this PR into 2 commits: adding yourself to list off maintainers and the package init?

aanderse · 2019-08-02T00:43:10Z

ping @bjornfor @infinisil @FRidh for next steps on this.

infinisil

Ideally a separate commit for every package. Other than that this looks good to merge to me, nice work :)

infinisil · 2019-08-02T01:24:06Z

@GrahamcOfBorg build scylladb

workflow · 2019-08-02T13:06:03Z

Hmm CI seems to fail, but I can't find any useful output from the borgs. Will verify building it on nixos, traveling right now and won't have access to my remote builder until Tuesday.

It builds fine here on my Ubuntu box, but that doesn't really help without sandboxing.

workflow · 2019-08-09T17:30:35Z

@infinisil Done rebasing, thanks! Tried the build on a couple of machines, including NixOS, and it builds fine (as long as it doesn't run out of memory, which can happen when using a lot of cores)

workflow · 2019-08-09T17:30:54Z

@GrahamcOfBorg build scylladb

infinisil

Commits are still a bit messed up. The first one is good, but the second one includes parts that should be in others. Each commit should be self-contained, this means for new packages this should include the addition in all-packages.nix.

infinisil · 2019-08-09T18:24:01Z

pkgs/development/tools/parsing/antlr/3.5.nix

  version = "3.5.2";
-  src = fetchurl {
-    url ="https://www.antlr3.org/download/antlr-${version}-complete.jar";
+  jar = fetchurl {


What's up with these antlr changes?

This includes the antlr libraries in the package output, which are not currently present in antlr 3.5.nix

My initial approach was to add a new package libantlr3cpp that exposes the libs only, and @aanderse suggested to modify the existing package instead!

I'll separate the change out into its own commit

workflow · 2019-08-10T11:01:57Z

Done and rebased.

@infinisil quite an attention to detail :)

infinisil · 2019-08-10T13:25:12Z

Hehe yeah, I kind of hate to do it, but it's better for the long-term quality of nixpkgs, since people learn from it and can do better PR's from the start in the future

That said, I have another complaint: The commit ordering needs to be adjusted, because in the current ordering, thrift-0_10 is used in the second commit, when it's only introduced in the 5th one (meaning scylladb will fail to even evaluate in the second commit).

So the ordering should be something like

maintainer-list
libsystemtap, thrift and antlr
scylladb

workflow · 2019-08-10T18:15:01Z

Haha, I actually noticed that as I rebased, and hoped to get away with it ;)
Not with this eagle-eye.

Thank you @infinisil, appreciated!

Reordered.

workflow · 2019-08-10T18:17:55Z

Weird, Github still shows the old order here, but it's correctly reordered downstream: https://github.com/tenx-tech/nixpkgs/commits/scylladb-upstream

infinisil · 2019-08-10T19:06:44Z

Yeah GitHub unfortunately has this bug (No it's not a feature, GitHub!)

I can see the order is correct with git log, thanks :)

infinisil requested changes May 16, 2019

View reviewed changes

workflow force-pushed the scylladb-upstream branch from 4934753 to 1e28044 Compare May 19, 2019 11:55

ofborg bot added the 8.has: package (new) label May 19, 2019

ofborg bot requested a review from bjornfor May 19, 2019 12:48

ofborg bot added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels May 19, 2019

aanderse requested changes Jul 17, 2019

View reviewed changes

FRidh requested changes Jul 17, 2019

View reviewed changes

ofborg bot added the 2.status: merge conflict label Jul 24, 2019

ofborg bot removed the 2.status: merge conflict label Jul 24, 2019

infinisil approved these changes Aug 2, 2019

View reviewed changes

workflow force-pushed the scylladb-upstream branch from 6da7d42 to de9b354 Compare August 9, 2019 17:27

infinisil reviewed Aug 9, 2019

View reviewed changes

workflow force-pushed the scylladb-upstream branch from de9b354 to 8df4e41 Compare August 10, 2019 11:00

workflow added 4 commits August 11, 2019 02:11

Add farlion to maintainer list

56dc124

antlr: add library outputs to 3.5.2

cb5b65c

libsystemtap: init at 3.2

b0a4801

thrift: re-add compatibility 0.10

53a4c35

scylladb: init at 3.0.5

72330fc

workflow force-pushed the scylladb-upstream branch from 8df4e41 to 72330fc Compare August 10, 2019 18:11

infinisil approved these changes Aug 10, 2019

View reviewed changes

infinisil merged commit 67600d7 into NixOS:master Aug 10, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

scylladb: init at 3.0.5 #61438

scylladb: init at 3.0.5 #61438

workflow commented May 13, 2019 •

edited

Loading

workflow commented May 19, 2019

nixos-discourse commented Jul 17, 2019

aanderse left a comment

aanderse Jul 17, 2019

workflow Jul 23, 2019

aanderse Jul 23, 2019

workflow Jul 23, 2019

workflow Jul 24, 2019

aanderse Jul 17, 2019

workflow Jul 23, 2019

workflow Jul 23, 2019 •

edited

Loading

FRidh Jul 17, 2019

workflow Jul 25, 2019

workflow commented Jul 24, 2019

workflow commented Jul 25, 2019 •

edited

Loading

aanderse commented Aug 2, 2019

aanderse commented Aug 2, 2019

infinisil left a comment •

edited

Loading

infinisil commented Aug 2, 2019

workflow commented Aug 2, 2019

workflow commented Aug 9, 2019

workflow commented Aug 9, 2019

infinisil left a comment

infinisil Aug 9, 2019

workflow Aug 10, 2019 •

edited

Loading

workflow commented Aug 10, 2019

infinisil commented Aug 10, 2019

workflow commented Aug 10, 2019

workflow commented Aug 10, 2019

infinisil commented Aug 10, 2019

scylladb: init at 3.0.5 #61438

scylladb: init at 3.0.5 #61438

Conversation

workflow commented May 13, 2019 • edited Loading

Motivation for this change

Things done

workflow commented May 19, 2019

nixos-discourse commented Jul 17, 2019

aanderse left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

workflow Jul 23, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

workflow commented Jul 24, 2019

workflow commented Jul 25, 2019 • edited Loading

aanderse commented Aug 2, 2019

aanderse commented Aug 2, 2019

infinisil left a comment • edited Loading

Choose a reason for hiding this comment

infinisil commented Aug 2, 2019

workflow commented Aug 2, 2019

workflow commented Aug 9, 2019

workflow commented Aug 9, 2019

infinisil left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

workflow Aug 10, 2019 • edited Loading

Choose a reason for hiding this comment

workflow commented Aug 10, 2019

infinisil commented Aug 10, 2019

workflow commented Aug 10, 2019

workflow commented Aug 10, 2019

infinisil commented Aug 10, 2019

workflow commented May 13, 2019 •

edited

Loading

workflow Jul 23, 2019 •

edited

Loading

workflow commented Jul 25, 2019 •

edited

Loading

infinisil left a comment •

edited

Loading

workflow Aug 10, 2019 •

edited

Loading