protobuf3: support cross compilation #48450

ryan4729 · 2018-10-15T07:45:00Z

Provide a native c compiler, which, and protoc

Motivation for this change

Support cross compilation for protobuf3. For this to work we need to:

ensure a native c compiler is on the PATH (for jsembed)
ensure a native which is on the PATH (for the configure script to find the native c compiler)
provide a native protoc binary as a configure flag (used to generate code at build-time)

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions (gentoo with nix-2.0.4)
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

Provide a native c compiler, which, and protoc

lheckemann

In principle, LGTM. Not sure I like this externalProtoc though — it took reading almost the whole diff for me to realise it was a boolean and not a reference to some external protoc. While this could be fixed by just changing the name to e.g. useExternalProtoc (which indicates the booleanness better) I think a better approach might be to just use (stdenv.hostPlatform != stdenv.buildPlatform) in generic-v3.nix (maybe with a let binding if you need to use it in multiple locations) directly. This results in much less cruft in all-packages.nix (in fact allowing it to remain completely as is) and avoids the need to thread the variables through in protobuf/3.*.nix.

lheckemann · 2018-10-15T08:19:45Z

pkgs/development/libraries/protobuf/3.1.nix


 callPackage ./generic-v3.nix {
  version = "3.1.0";
  sha256 = "0qlvpsmqgh9nw0k4zrxlxf75pafi3p0ahz99v6761b903y8qyv4i";
+  externalProtoc = externalProtoc;
+  buildProtobuf = buildProtobuf;


You could use inherit externalProtoc buildProtobuf; instead here.

lheckemann · 2018-10-15T08:28:10Z

@GrahamcOfBorg build pkgsCross.aarch64-multiplatform.protobuf3_6

ryan4729 · 2018-10-15T10:08:11Z

I need to be able to select the specific matching version (protobuf3_5, protobuf3_6, etc) from generic-v3.nix so I can't use a hardcoded attribute path for the native package. I changed it to a function that takes buildProtobuf and stdenv as parameters, and then passes in a native version for the buildProtobuf only for a cross-compile. This de-crufts the rest of the files.

lheckemann · 2018-10-15T10:08:12Z

pkgs/development/libraries/protobuf/generic-v3.nix

@@ -31,10 +30,10 @@ stdenv.mkDerivation rec {
  '';

  nativeBuildInputs = [ autoreconfHook buildPackages.which buildPackages.stdenv.cc ]
-                      ++ stdenv.lib.optional useExternalProtoc [ buildProtobuf ];
+                      ++ (if buildProtobuf == null then [] else [ buildProtobuf ]);


buildInputs entries which are null will just be ignored, so you don't need to put this condition here.

lheckemann · 2018-10-15T10:10:40Z

pkgs/development/libraries/protobuf/generic-v3.nix

 , version, sha256
 , ...
 }:

-stdenv.mkDerivation rec {
+let
+mkProtobufDerivation = buildProtobuf: stdenv: stdenv.mkDerivation rec {


I don't think you need a function here? Can't you just do let buildProtobuf = if stdenv.hostPlatform != stdenv.buildPlatform then buildPackages.protobuf else null; in stdenv.mkDerivation rec {…}? I'm not sure about the exact recursion implications of that, but I think it should work.

I can't do this because the host and build protobuf versions must match. buildPackages.protobuf is defined in all-packages.nix as protobuf = protobuf3_6;. So using buildPackages.protobuf, when I try to cross-compile e.g. protobuf3_1 I see that it wants to build protobuf3_6 for the build version:

these derivations will be built: /nix/store/cszcilxncv2hhxb5n5j41y1bxbxbwcb6-protobuf-3.6.1.drv /nix/store/lp6xfnwlli29r24vb9pmrgmnd84zbd7m-protobuf-3.1.0-aarch64-unknown-linux-gnu.drv

The protobuf docs say they want the versions to match in the cross compile: https://github.com/protocolbuffers/protobuf/blob/master/src/README.md:

Either way, you must make sure that the protoc executable you use has the same version as the protobuf source code you are trying to use it with.

lheckemann · 2018-10-15T12:19:13Z

@GrahamcOfBorg build pkgsCross.aarch64-multiplatform.protobuf3_6

GrahamcOfBorg · 2018-10-15T12:20:01Z

Failure on x86_64-darwin (full log)

Attempted: pkgsCross.aarch64-multiplatform.protobuf3_6

Partial log (click to expand)

builder for '/nix/store/ryphibmpkybq5jiixwsbp7gis1w0qkpp-linux-headers-4.18.3-aarch64-unknown-linux-gnu.drv' failed with exit code 2
cannot build derivation '/nix/store/ga9fig6p18j8k3d5cpc4x0lpk1mgidcs-glibc-2.27-aarch64-unknown-linux-gnu.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/0jzfhiaiajkg68blk38l9xk55snwf690-aarch64-unknown-linux-gnu-binutils-wrapper-2.30.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/2adj4rjbmc124094ybj8w733r64zqwf9-aarch64-unknown-linux-gnu-stage-final-gcc-debug-7.3.0.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/qgivv901jhnpaa8xczfpbaq0pwr7mw0p-aarch64-unknown-linux-gnu-stage-final-gcc-debug-wrapper-7.3.0.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/bfb5h7di349m96lg1h2vpmnixjf8mdmb-stdenv-darwin.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/wdy5jprnz782pdlj0c400vm514wiybv3-zlib-1.2.11-aarch64-unknown-linux-gnu.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/2mbk25bgpn88v7rip0vhppj7lc7vg0n2-protobuf-3.6.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/41mldzd050s1g2avghn9y1x9liwip1rl-protobuf-3.6.1-aarch64-unknown-linux-gnu.drv': 3 dependencies couldn't be built
error: build of '/nix/store/41mldzd050s1g2avghn9y1x9liwip1rl-protobuf-3.6.1-aarch64-unknown-linux-gnu.drv' failed

GrahamcOfBorg · 2018-10-15T12:21:10Z

Success on aarch64-linux (full log)

Attempted: pkgsCross.aarch64-multiplatform.protobuf3_6

Partial log (click to expand)

shrinking RPATHs of ELF executables and libraries in /nix/store/8s1rh9c02h3727v0y9y6lnha60pbzwqa-protobuf-3.6.1
shrinking /nix/store/8s1rh9c02h3727v0y9y6lnha60pbzwqa-protobuf-3.6.1/bin/protoc
shrinking /nix/store/8s1rh9c02h3727v0y9y6lnha60pbzwqa-protobuf-3.6.1/lib/libprotoc.so.17.0.0
shrinking /nix/store/8s1rh9c02h3727v0y9y6lnha60pbzwqa-protobuf-3.6.1/lib/libprotobuf.so.17.0.0
shrinking /nix/store/8s1rh9c02h3727v0y9y6lnha60pbzwqa-protobuf-3.6.1/lib/libprotobuf-lite.so.17.0.0
strip is /nix/store/p9akxn2sfy4wkhqdqa3li97pc6jaz3r1-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/8s1rh9c02h3727v0y9y6lnha60pbzwqa-protobuf-3.6.1/lib  /nix/store/8s1rh9c02h3727v0y9y6lnha60pbzwqa-protobuf-3.6.1/bin
patching script interpreter paths in /nix/store/8s1rh9c02h3727v0y9y6lnha60pbzwqa-protobuf-3.6.1
checking for references to /build in /nix/store/8s1rh9c02h3727v0y9y6lnha60pbzwqa-protobuf-3.6.1...
/nix/store/8s1rh9c02h3727v0y9y6lnha60pbzwqa-protobuf-3.6.1

GrahamcOfBorg · 2018-10-15T12:27:30Z

Success on x86_64-linux (full log)

Attempted: pkgsCross.aarch64-multiplatform.protobuf3_6

Partial log (click to expand)

shrinking RPATHs of ELF executables and libraries in /nix/store/faxai1ak2vi8fli5561gsljiy3nn0mgw-protobuf-3.6.1-aarch64-unknown-linux-gnu
shrinking /nix/store/faxai1ak2vi8fli5561gsljiy3nn0mgw-protobuf-3.6.1-aarch64-unknown-linux-gnu/bin/protoc
shrinking /nix/store/faxai1ak2vi8fli5561gsljiy3nn0mgw-protobuf-3.6.1-aarch64-unknown-linux-gnu/lib/libprotobuf-lite.so.17.0.0
shrinking /nix/store/faxai1ak2vi8fli5561gsljiy3nn0mgw-protobuf-3.6.1-aarch64-unknown-linux-gnu/lib/libprotobuf.so.17.0.0
shrinking /nix/store/faxai1ak2vi8fli5561gsljiy3nn0mgw-protobuf-3.6.1-aarch64-unknown-linux-gnu/lib/libprotoc.so.17.0.0
aarch64-unknown-linux-gnu-strip is /nix/store/l7qxjg8gclh4qm2k691abvhsah79pym7-aarch64-unknown-linux-gnu-binutils-2.30/bin/aarch64-unknown-linux-gnu-strip
stripping (with command aarch64-unknown-linux-gnu-strip and flags -S) in /nix/store/faxai1ak2vi8fli5561gsljiy3nn0mgw-protobuf-3.6.1-aarch64-unknown-linux-gnu/lib  /nix/store/faxai1ak2vi8fli5561gsljiy3nn0mgw-protobuf-3.6.1-aarch64-unknown-linux-gnu/bin
patching script interpreter paths in /nix/store/faxai1ak2vi8fli5561gsljiy3nn0mgw-protobuf-3.6.1-aarch64-unknown-linux-gnu
checking for references to /build in /nix/store/faxai1ak2vi8fli5561gsljiy3nn0mgw-protobuf-3.6.1-aarch64-unknown-linux-gnu...
/nix/store/faxai1ak2vi8fli5561gsljiy3nn0mgw-protobuf-3.6.1-aarch64-unknown-linux-gnu

lheckemann · 2018-10-15T12:31:58Z

Whoops, merged this too hastily. As lots of things depend on protobuf, this is a mass-rebuild and should be based on the staging branch.

PR #48450

lheckemann · 2018-10-15T12:39:28Z

Pushed to staging as 7414ec1.

protobuf3: support cross compilation

3653909

Provide a native c compiler, which, and protoc

GrahamcOfBorg added 10.rebuild-darwin: 101-500 10.rebuild-linux: 101-500 labels Oct 15, 2018

lheckemann requested changes Oct 15, 2018

View reviewed changes

This comment has been minimized.

Sign in to view

protobuf3: fix parameter naming

12ededf

This comment has been minimized.

Sign in to view

protobuf3: clean up cross compile

1f9188b

lheckemann reviewed Oct 15, 2018

View reviewed changes

protobuf3: remove unnecessary null check

65a81af

lheckemann merged commit dc6abb3 into NixOS:master Oct 15, 2018

lheckemann mentioned this pull request Oct 15, 2018

Revert "protobuf3: support cross compilation" #48461

Merged

lheckemann pushed a commit that referenced this pull request Oct 15, 2018

protobuf3: support cross compilation

7414ec1

PR #48450

ryan4729 deleted the proto3-cross branch October 15, 2018 16:09

flokli mentioned this pull request Mar 31, 2019

vlc: Add chromecast support; libmicrodns: Init at 0.0.10 #58588

Merged

10 tasks

Janik-Haag added the 12. first-time contribution label Jun 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

protobuf3: support cross compilation #48450

protobuf3: support cross compilation #48450

ryan4729 commented Oct 15, 2018 •

edited

lheckemann left a comment

lheckemann Oct 15, 2018

lheckemann commented Oct 15, 2018

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ryan4729 commented Oct 15, 2018

lheckemann Oct 15, 2018

lheckemann Oct 15, 2018

ryan4729 Oct 15, 2018 •

edited

lheckemann Oct 15, 2018

lheckemann commented Oct 15, 2018

GrahamcOfBorg commented Oct 15, 2018

GrahamcOfBorg commented Oct 15, 2018

GrahamcOfBorg commented Oct 15, 2018

lheckemann commented Oct 15, 2018

lheckemann commented Oct 15, 2018

protobuf3: support cross compilation #48450

protobuf3: support cross compilation #48450

Conversation

ryan4729 commented Oct 15, 2018 • edited

Motivation for this change

Things done

lheckemann left a comment

Choose a reason for hiding this comment

lheckemann Oct 15, 2018

Choose a reason for hiding this comment

lheckemann commented Oct 15, 2018

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ryan4729 commented Oct 15, 2018

lheckemann Oct 15, 2018

Choose a reason for hiding this comment

lheckemann Oct 15, 2018

Choose a reason for hiding this comment

ryan4729 Oct 15, 2018 • edited

Choose a reason for hiding this comment

lheckemann Oct 15, 2018

Choose a reason for hiding this comment

lheckemann commented Oct 15, 2018

GrahamcOfBorg commented Oct 15, 2018

GrahamcOfBorg commented Oct 15, 2018

GrahamcOfBorg commented Oct 15, 2018

lheckemann commented Oct 15, 2018

lheckemann commented Oct 15, 2018

ryan4729 commented Oct 15, 2018 •

edited

ryan4729 Oct 15, 2018 •

edited