Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: NixOS/nixpkgs
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 894a9994e86c
Choose a base ref
...
head repository: NixOS/nixpkgs
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: 0f7a562eddb5
Choose a head ref
  • 5 commits
  • 4 files changed
  • 1 contributor

Commits on Nov 17, 2018

  1. openjdk11: enable ZGC on x86_64-linux

    The Z Garbage Collector is a concurrent, scalable, low latency garbage
    collector designed to meet extremely-low-pause-time requirements for
    small-to-multi-TB heap sizes.
    
    ZGC can be enabled with the magical incantation:
    
        $ java -XX:+UnlockExperimentalVMOptions -XX:+UseZGC ...
    
    Currently, ZGC is only available for x86_64-linux (though a port for
    aarch64-linux may become available at a future time.) There are also a
    number of other features that currently aren't present, such as JVMCI
    integration (meaning compiler tools like Graal which require JVMCI will
    not work with ZGC enabled.)
    
    Signed-off-by: Austin Seipp <aseipp@pobox.com>
    (cherry picked from commit 1629147)
    thoughtpolice committed Nov 17, 2018
    2
    Copy the full SHA
    5d0ef3f View commit details
  2. foundationdb60: 6.0.4pre2497_73d64cb2 -> 6.0.11pre2716_9e8c1941e

    Signed-off-by: Austin Seipp <aseipp@pobox.com>
    (cherry picked from commit f1737fa)
    thoughtpolice committed Nov 17, 2018

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    1d3d52a View commit details
  3. foundationdb: include fdb.options in .dev for binding generators

    Signed-off-by: Austin Seipp <aseipp@pobox.com>
    (cherry picked from commit 32948a6)
    thoughtpolice committed Nov 17, 2018
    Copy the full SHA
    eec1fa5 View commit details
  4. foundationdb: rework python bindings, build system

    FoundationDB uses Python at build time for some code generation.
    However, it also has the official python bindings inside the source code
    too, and the code for the Python bindings has some of it auto-generated
    at compile time.
    
    This made building python packages unattractive: we want to use the
    source code generated from the FoundationDB build, but we don't want to
    rebuild it. Previously we would override the 'python' input to the
    FoundationDB module, but this meant we would do a complete rebuild, as
    it was a necessary build time dependency, even though the resulting
    generated code itself would not change. Furthermore, FoundationDB
    versions < 6.0 don't properly support Python 3 *for the build system*,
    though the bindings supported it, so that caused build failures. But the
    first effect is the worst: it meant building separate python2 and
    python3 packages implied two complete rebuilds of a single FoundationDB
    version. This meant rather than 3 FDB builds, we'd do 3*N where N = the
    number of major Python versions we support.
    
    Finally, because we did not use pip to generate a wheel that we install
    with metadata recorded for the installation, the FoundationDB python
    package couldn't be used as an input to other setup.py-based packages:
    there would be no recorded metadata in the dist-info folder which would
    say this is the foundationdb package. This greatly limits its utility.
    
    To fix all this, we do a few things:
    
      - Apply some patches to fix the build system with Python 3.x for
        older FoundationDB versions. (This is nice if end-users have
        overridden the global Python version for some reason.)
      - Move python directly into nativeBuildInputs, so it is only a
        build time dependency.
      - Take the python source code from the ./bindings directory and
        tar it up use later after the build is done, so we get to keep
        the generated code. This is the new 'pythonsrc' output from the
        build. This code doesn't change based on whether or not the input
        or resulting package is using Python 2 or 3, it's totally
        deterministic.
      - The build system also patches up the python source code a little,
        so it can be installed directly with setup.py (it needs a little
        stuff that it normally expects the build system to do.)
      - Rework the python package to a separate file that uses
        buildPythonPackage directly. Because the source code is already
        prepared, it needs almost nothing else. Furthermore, this kills
        the override itself for the foundationdb package, meaning rebuilds
        are no longer needed.
      - This package is very simple and just uses foundationdb.pythonsrc
        as its source input. It also ensures a link to libfdb_c.so can
        be found by ctypes (using substituteInPlace)
      - python-packages.nix now just uses callPackage directly.
    
    The net effect of this is, most importantly, that python packages do not
    imply a full rebuild of the server source code: building python2 and
    python3 packages from a version of FoundationDB now does not need to
    override the foundationdb python input, reducing the number of needless
    builds. They instead just run setup.py with the given version as input.
    
    The second biggest effect is that wheel metadata is recorded correctly,
    meaning dependent-python-packages that want to use the FoundationDB
    bindings e.g. from PyPi should now work fine with buildPythonPackage.
    
    Signed-off-by: Austin Seipp <aseipp@pobox.com>
    (cherry picked from commit 6054dab)
    thoughtpolice committed Nov 17, 2018
    Copy the full SHA
    d359c39 View commit details
  5. foundationdb60: 6.0.11pre2716 -> 6.0.15

    Signed-off-by: Austin Seipp <aseipp@pobox.com>
    (cherry picked from commit 8d4f8a6)
    thoughtpolice committed Nov 17, 2018
    Copy the full SHA
    0f7a562 View commit details
Showing with 59 additions and 24 deletions.
  1. +1 −0 pkgs/development/compilers/openjdk/10.nix
  2. +31 −13 pkgs/servers/foundationdb/default.nix
  3. +24 −0 pkgs/servers/foundationdb/python.nix
  4. +3 −11 pkgs/top-level/python-packages.nix
1 change: 1 addition & 0 deletions pkgs/development/compilers/openjdk/10.nix
Original file line number Diff line number Diff line change
@@ -69,6 +69,7 @@ let
# See https://www.mail-archive.com/openembedded-devel@lists.openembedded.org/msg49006.html
"--with-extra-cflags=-Wno-error=deprecated-declarations -Wno-error=format-contains-nul -Wno-error=unused-result"
''
+ lib.optionalString (architecture == "amd64") "\"--with-jvm-features=zgc\""
+ lib.optionalString minimal "\"--enable-headless-only\""
+ ");"
# https://bugzilla.redhat.com/show_bug.cgi?id=1306558
44 changes: 31 additions & 13 deletions pkgs/servers/foundationdb/default.nix
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{ stdenv49
, lib, fetchurl, fetchFromGitHub
, lib, fetchurl, fetchpatch, fetchFromGitHub

, which, findutils, m4, gawk
, python, openjdk, mono58, libressl
@@ -47,8 +47,8 @@ let
inherit rev sha256;
};

nativeBuildInputs = [ gawk which m4 findutils mono58 ];
buildInputs = [ python openjdk libressl boost ];
nativeBuildInputs = [ python openjdk gawk which m4 findutils mono58 ];
buildInputs = [ libressl boost ];

patches =
[ # For 5.2+, we need a slightly adjusted patch to fix all the ldflags
@@ -57,10 +57,24 @@ let
then ./ldflags-6.0.patch
else ./ldflags-5.2.patch)
else ./ldflags-5.1.patch)
] ++
]
# for 6.0+, we do NOT need to apply this version fix, since we can specify
# it ourselves. see configurePhase
(lib.optional (!lib.versionAtLeast version "6.0") ./fix-scm-version.patch);
++ (lib.optional (!lib.versionAtLeast version "6.0") ./fix-scm-version.patch)
# Versions less than 6.0 have a busted Python 3 build due to an outdated
# use of 'print'. Also apply an update to the six module with many bugfixes,
# which is in 6.0+ as well
++ (lib.optional (!lib.versionAtLeast version "6.0") (fetchpatch {
name = "update-python-six.patch";
url = "https://github.com/apple/foundationdb/commit/4bd9efc4fc74917bc04b07a84eb065070ea7edb2.patch";
sha256 = "030679lmc86f1wzqqyvxnwjyfrhh54pdql20ab3iifqpp9i5mi85";
}))
++ (lib.optional (!lib.versionAtLeast version "6.0") (fetchpatch {
name = "import-for-python-print.patch";
url = "https://github.com/apple/foundationdb/commit/ded17c6cd667f39699cf663c0e87fe01e996c153.patch";
sha256 = "11y434w68cpk7shs2r22hyrpcrqi8vx02cw7v5x79qxvnmdxv2an";
}))
;

postPatch = ''
# note: this does not do anything for 6.0+
@@ -106,7 +120,6 @@ let

installPhase = ''
mkdir -vp $out/{bin,libexec/plugins} $lib/{lib,share/java} $dev/include/foundationdb
mkdir -vp $python/lib/${python.libPrefix}/site-packages
'' + lib.optionalString (!lib.versionAtLeast version "6.0") ''
# we only copy the TLS library on < 6.0, since it's compiled-in otherwise
@@ -117,15 +130,21 @@ let
cp -v ./lib/libfdb_c.so $lib/lib
cp -v ./bindings/c/foundationdb/fdb_c.h $dev/include/foundationdb
cp -v ./bindings/c/foundationdb/fdb_c_options.g.h $dev/include/foundationdb
cp -v ./fdbclient/vexillographer/fdb.options $dev/include/foundationdb
# java
cp -v ./bindings/java/foundationdb-client.jar $lib/share/java/fdb-java.jar
# python
cp LICENSE ./bindings/python
substitute ./bindings/python/setup.py.in ./bindings/python/setup.py \
--replace 'VERSION' "${version}"
rm -f ./bindings/python/setup.py.in
rm -f ./bindings/python/fdb/*.pth # remove useless files
cp -R ./bindings/python/fdb $python/lib/${python.libPrefix}/site-packages/fdb
# symlink a copy of the shared object into place, so that impl.py can load it
ln -sv $lib/lib/libfdb_c.so $python/lib/${python.libPrefix}/site-packages/fdb/libfdb_c.so
rm -f ./bindings/python/*.rst ./bindings/python/*.mk
cp -R ./bindings/python/ tmp-pythonsrc/
tar -zcf $pythonsrc --transform s/tmp-pythonsrc/python-foundationdb/ ./tmp-pythonsrc/
# binaries
for x in fdbbackup fdbcli fdbserver fdbmonitor; do
@@ -139,7 +158,7 @@ let
ln -sfv $out/bin/fdbbackup $out/libexec/backup_agent
'';

outputs = [ "out" "lib" "dev" "python" ];
outputs = [ "out" "lib" "dev" "pythonsrc" ];

meta = with stdenv.lib; {
description = "Open source, distributed, transactional key-value store";
@@ -165,9 +184,8 @@ in with builtins; {
};

foundationdb60 = makeFdb rec {
version = "6.0.4pre2497_${substring 0 8 rev}";
version = "6.0.15";
branch = "release-6.0";
rev = "73d64cb244714c19bcc651122f6e7a9236aa11b5";
sha256 = "1jzmrf9kj0brqddlmxvzhj27r6843790jnqwkv1s3ri21fqb3hs7";
sha256 = "1z8104nj1qn738bs1zjiq1mdn8dnj4vksb3fh503mf3ygl54mjbw";
};
}
24 changes: 24 additions & 0 deletions pkgs/servers/foundationdb/python.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
{ buildPythonPackage, lib, foundationdb }:

buildPythonPackage rec {
pname = "foundationdb";
version = foundationdb.version;

src = foundationdb.pythonsrc;
unpackCmd = "tar xf $curSrc";

patchPhase = ''
substituteInPlace ./fdb/impl.py \
--replace libfdb_c.so "${foundationdb.lib}/lib/libfdb_c.so"
'';

doCheck = false;

meta = with lib; {
description = "Python bindings for FoundationDB";
homepage = https://www.foundationdb.org;
license = with licenses; [ asl20 ];
maintainers = with maintainers; [ thoughtpolice ];
};
}

14 changes: 3 additions & 11 deletions pkgs/top-level/python-packages.nix
Original file line number Diff line number Diff line change
@@ -17324,17 +17324,9 @@ EOF

rfc7464 = callPackage ../development/python-modules/rfc7464 { };

foundationdb51 = (toPythonModule (pkgs.fdbPackages.override {
inherit python;
}).foundationdb51).python;

foundationdb52 = (toPythonModule (pkgs.fdbPackages.override {
inherit python;
}).foundationdb52).python;

foundationdb60 = (toPythonModule (pkgs.fdbPackages.override {
inherit python;
}).foundationdb60).python;
foundationdb51 = callPackage ../servers/foundationdb/python.nix { foundationdb = pkgs.foundationdb51; };
foundationdb52 = callPackage ../servers/foundationdb/python.nix { foundationdb = pkgs.foundationdb52; };
foundationdb60 = callPackage ../servers/foundationdb/python.nix { foundationdb = pkgs.foundationdb60; };

libtorrentRasterbar = (toPythonModule (pkgs.libtorrentRasterbar.override {
inherit python;