Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Carnix: 0.6.1 -> 0.6.4 #34363

Closed
wants to merge 3 commits into from
Closed

Carnix: 0.6.1 -> 0.6.4 #34363

wants to merge 3 commits into from

Conversation

P-E-Meunier
Copy link
Contributor

Motivation for this change

This new version of Carnix makes it easier to call the generated packages, by:

  • dropping the requirement to specify the package version in the name, and
  • accepting a set of features.
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

@grahamc
Copy link
Member

grahamc commented Jan 28, 2018

@GrahamcOfBorg build carnix

Copy link

@GrahamcOfBorg GrahamcOfBorg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Success on x86_64-linux

Partial log (click to expand)

no configure script, doing nothing
building
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/yy0nax0hjndznl8cl0h8lv2h67k94svf-rust_carnix-0.6.4
strip is /nix/store/mdyy001q67hiks0g24ra53z7ckm4jfr4-binutils-2.28.1/bin/strip
stripping (with command strip and flags -S) in /nix/store/yy0nax0hjndznl8cl0h8lv2h67k94svf-rust_carnix-0.6.4/lib 
patching script interpreter paths in /nix/store/yy0nax0hjndznl8cl0h8lv2h67k94svf-rust_carnix-0.6.4
checking for references to /tmp/nix-build-rust_carnix-0.6.4.drv-0 in /nix/store/yy0nax0hjndznl8cl0h8lv2h67k94svf-rust_carnix-0.6.4...
/nix/store/yy0nax0hjndznl8cl0h8lv2h67k94svf-rust_carnix-0.6.4

Full log

Copy link

@GrahamcOfBorg GrahamcOfBorg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Success on x86_64-darwin

Partial log (click to expand)

patching sources
configuring
no configure script, doing nothing
building
installing
post-installation fixup
strip is /nix/store/i1zz228nl3bljcbb83fq0rq7lr7nvv4y-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/pmbqgh6d3mx2wvhlglq18c7kslgl6yll-rust_carnix-0.6.4/lib
patching script interpreter paths in /nix/store/pmbqgh6d3mx2wvhlglq18c7kslgl6yll-rust_carnix-0.6.4
/nix/store/pmbqgh6d3mx2wvhlglq18c7kslgl6yll-rust_carnix-0.6.4

Full log

Copy link

@GrahamcOfBorg GrahamcOfBorg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Failure on aarch64-linux (full log)

Partial log (click to expand)

cannot build derivation '/nix/store/81id9xzrj1gx54smwcyx1g08fqd7yb4m-rust_syn-0.11.11.drv': 4 dependencies couldn't be built
cannot build derivation '/nix/store/6pvm5bmqkyw5zf0x2c8c73pkaggmhbrg-rust_tempdir-0.3.5.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/m0iliq3gfckgd8bycywizxygf2xhj7zz-rust_thread_local-0.3.4.drv': 4 dependencies couldn't be built
cannot build derivation '/nix/store/25793q0cfilclqw4dnyl6vniaril7f4a-rust_error-chain-0.11.0.drv': 6 dependencies couldn't be built
cannot build derivation '/nix/store/73ia5k6fig5fz13n635py6p758kblgh1-rust_regex-0.2.2.drv': 10 dependencies couldn't be built
cannot build derivation '/nix/store/nslwkl7vdjvskyrivmbfhd0id6pjyla6-rust_serde_derive_internals-0.17.0.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/zgl68mb8ggga9nmvpfzq8cx1fkpgni9f-rust_env_logger-0.4.3.drv': 12 dependencies couldn't be built
cannot build derivation '/nix/store/64jzkqc99l0di6zi9k3s84zzdgb957wb-rust_serde_derive-1.0.21.drv': 6 dependencies couldn't be built
cannot build derivation '/nix/store/3bh14aal3rsdxx9211mklja3sgz6jbcn-rust_carnix-0.6.4.drv': 48 dependencies couldn't be built
error: build of '/nix/store/3bh14aal3rsdxx9211mklja3sgz6jbcn-rust_carnix-0.6.4.drv' failed

crateName = "carnix";
version = "0.6.0";
version = "0.6.4";
authors = [ "pe@pijul.org <pe@pijul.org>" ];
src = include [ "Cargo.toml" "src/main.rs" "src/cache.rs" "src/cfg.rs" "src/krate/mod.rs" "src/krate/prefetch.rs" ] ./.;
Copy link

@ghost ghost Jan 29, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it should be

    sha256 = "0r7vk1xdfdnp4yi4fy3dmjc992ys8qr0mh8wyvwi2h2sdpm2mfbs";

insted of

    src = ...;

@boxofrox
Copy link
Contributor

boxofrox commented Feb 2, 2018

The changes to features in carnix 0.6.4 cause an error in my Rust project when building the num-complex crate as an indirect dependency.

project
 |
 +-- chrono
 |    |
 |    `-- num
 |         |
 |         `-- num-complex
 |
 `-- ndarray
      |
      `-- num-complex

Build error with Carnix 0.6.4 (Click to expand)
Building src/lib.rs (num-complex)
Running rustc --crate-name num_complex src/lib.rs --crate-type lib -C opt-level=3 -C metadata=b465201f42 -C extra-filename=-b465201f42 --cfg feature="rustc-serialize" --cfg feature="default" --out-dir target/lib --emit=dep-info,link -L dependency=target/deps --extern num_traits=/nix/store/8j3a9q8gjsf4l5iyn84x72ycnj9zdpwj-rust_num-traits-0.1.42/lib/libnum_traits-cd53366641.rlib --cap-lints allow --color always
error[E0658]: use of unstable library feature 'rustc_private': this crate is being loaded from the sysroot, an unstable location; did you mean to load this crate from crates.io via `Cargo.toml` instead? (see issue #27812)
  --> src/lib.rs:20:1
   |
20 | extern crate rustc_serialize;
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = help: add #![feature(rustc_private)] to the crate attributes to enable

error: aborting due to previous error

builder for ‘/nix/store/9ahmk3pwakplbsa7p851glb08fcg532i-rust_num-complex-0.1.41.drv’ failed with exit code 101
cannot build derivation ‘/nix/store/s95h8rhcpd6sz7hzw5s52mmy89vf89mw-rust_fuzzyd-0.13.0.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/s95h8rhcpd6sz7hzw5s52mmy89vf89mw-rust_fuzzyd-0.13.0.drv’ failed

Building my project with carnix 0.6.1 works fine, and 0.5.2 before that.

Build log for num_complex with carnix 0.6.1 (Click to expand)
Building src/lib.rs (num-complex)
Running rustc --crate-name num_complex src/lib.rs --crate-type lib -C opt-level=3 -C metadata=b465201f42 -C extra-filename=-b465201f42 --out-dir target/lib --emit=dep-info,link -L dependency=target/deps --extern num_traits=/nix/store/4jk3mkvgnbqamfnwng2gw01yy3pbr6dl-rust_num-traits-0.1.42/lib/libnum_traits-cd53366641.rlib --cap-lints allow --color always
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/nssfaramr4abs3xkhls1q4yqxwshlwj6-rust_num-complex-0.1.41
strip is /nix/store/mdyy001q67hiks0g24ra53z7ckm4jfr4-binutils-2.28.1/bin/strip
stripping (with command strip and flags -S) in /nix/store/nssfaramr4abs3xkhls1q4yqxwshlwj6-rust_num-complex-0.1.41/lib
patching script interpreter paths in /nix/store/nssfaramr4abs3xkhls1q4yqxwshlwj6-rust_num-complex-0.1.41
checking for references to /tmp/nix-build-rust_num-complex-0.1.41.drv-0 in /nix/store/nssfaramr4abs3xkhls1q4yqxwshlwj6-rust_num-complex-0.1.41...
building path(s) ‘/nix/store/a94q4jagrzq21xn43f0k3rs4f0w6xi24-rust_num-integer-0.1.35’
unpacking sources
unpacking source archive /nix/store/j7v77a00h3cn91w19i9jjl5zajix0n2h-num-integer-0.1.35.tar.gz
source root is num-integer-0.1.35.tar.gz
patching sources
configuring
no configure script, doing nothing
building

Comparing the rustc commands shows that 0.6.4 added --cfg feature="rustc-serialize".

- Running rustc --crate-name num_complex src/lib.rs --crate-type lib -C opt-level=3 -C metadata=b465201f42 -C extra-filename=-b465201f42 --out-dir target/lib --emit=dep-info,link -L dependency=target/deps --extern num_traits=/nix/store/4jk3mkvgnbqamfnwng2gw01yy3pbr6dl-rust_num-traits-0.1.42/lib/libnum_traits-cd53366641.rlib --cap-lints allow --color always
+ Running rustc --crate-name num_complex src/lib.rs --crate-type lib -C opt-level=3 -C metadata=b465201f42 -C extra-filename=-b465201f42 --cfg feature="rustc-serialize" --cfg feature="default" --out-dir target/lib --emit=dep-info,link -L dependency=target/deps --extern num_traits=/nix/store/8j3a9q8gjsf4l5iyn84x72ycnj9zdpwj-rust_num-traits-0.1.42/lib/libnum_traits-cd53366641.rlib --cap-lints allow --color always

I'm not sure which is doing the wrong thing:

  • 0.6.1 wrongly fails to enable a default feature and compiles, or
  • 0.6.4 wrongly enables a default feature and the build fails.

I don't want to hijack this PR with a bug discussion, so if there's a better place to discuss, I can open a github issue or pijul discussion or whatever else is preferred and create an SSCCE.

@Mic92
Copy link
Member

Mic92 commented Feb 3, 2018

I am bit worried about the file sizes that are getting bigger and bigger with each release of carnix:

$ wc -l carnix-0.6.4/Cargo.lock carnix.nix
551 Cargo.lock
1508 carnix.nix
$ wc -c carnix.nix Cargo.lock
72813 carnix.nix
24678 Cargo.lock

Out of experience I know, that git does not like large generated text files.
Also nix will take longer to evaluate.

backtrace_0_3_4_features = f:
builtins.deepSeq f (lib.lists.foldl' (features: fun: fun features)
(lib.attrsets.recursiveUpdate f (rec {
backtrace_0_3_4."addr2line" =
Copy link
Member

@Mic92 Mic92 Feb 3, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this equivalent to?

backtrace_0_3_4."addr2line" = f.backtrace_0_3_4."addr2line" or
  f.backtrace_0_3_4."gimli-symbolize" or
  backtrace_0_3_4."gimli-symbolize" or false;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be great, but it doesn't look like it's equivalent:

nix-repl> { a = false; }.a or { a = true; }.a
false

@P-E-Meunier
Copy link
Contributor Author

@Mic92: this is because rust "features" are normally handled by Cargo. The first versions of Carnix computed them once and for all, but this was incorrect because they might depend on the platform. The last release (0.6.4) is supposed to shrink them a little, at least in bytes if not in lines, by using more appropriate features of Nix.

Some more context: in Rust, crates can have "features", which are supposed to be monotonic (i.e. features can't conflict), and might depend on the platform, or on other features, or sometimes on features in their dependent crates. In addition to that, some crates also have default features, which is a list of features activated unless specified otherwise.

Carnix 0.6.0 used an ugly idea to compute features: each crate A wanting a feature F from a dependency B would cause carnix to add a line of the form A.F.from_B = true, or sometimes A.F.from_B = hasFeature A G if feature F depends on whether feature G of A is active.

Carnix 0.6.4 does a better job by collecting features for all crates by passing them to feature-computing functions, which in turn pass them over to the dependencies' feature-computing function.

@boxofrox: I'd try it out. Do you use any feature from chrono from your "project" crate?

ansi_term_0_10_2 = { features?(ansi_term_0_10_2_features {}) }: ansi_term_0_10_2_ {};
ansi_term_0_10_2_features = f:
builtins.deepSeq f (lib.lists.foldl' (features: fun: fun features)
(lib.attrsets.recursiveUpdate f (rec {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this whole thing become a new function instead of pasting this for every crate?

builtins.deepSeq f (lib.lists.foldl' (features: fun: fun features) (lib.attrsets.recursiveUpdate f)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea. I'll do it now.

@Mic92
Copy link
Member

Mic92 commented Feb 3, 2018

@P-E-Meunier I see. It also looks a more readable then older versions.

@boxofrox
Copy link
Contributor

boxofrox commented Feb 3, 2018

@boxofrox: I'd try it out. Do you use any feature from chrono from your "project" crate?

chrono 0.4.0 doesn't have any features to enable that I'm aware of. chrono does depend on the num crate, and disables default features in num [1], likewise, ndarray disables default features of num-complex [2].

However, num optionally depends on num-complex as a default feature or complex feature [3], but doesn't disable default features in num-complex. So, here we might have a conflict: num -> num-complex(default) and ndarray -> num-complex(no-default), but I'm pretty sure num shouldn't be pulling in num-complex as a dependency in my project, since chrono disabled default features of its num dependency chrono -> num(no-default) -> "no dep on num-complex".

I'll create an SSCCE on github and post a link.

[1]: https://github.com/chronotope/chrono/blob/master/Cargo.toml#L24
[2]: https://github.com/bluss/rust-ndarray/blob/master/Cargo.toml#L28
[3]: https://github.com/rust-num/num/blob/master/Cargo.toml#L46

Full Cargo.toml (Click to expand)
[package]
authors = ["Justin Charette <charetjc@gmail.com>"]
name = "fuzzyd"
version = "0.13.1"
[[bin]]
name = "fuzzyd"
path = "src/bin/fuzzyd/main.rs"
[[bin]]
name = "fuzzy-debug"
path = "src/bin/fuzzy-debug.rs"
[dependencies]
as_num = "0.2.3"
assert_matches = "1.1.0"
bytes = "0.4.3"
chrono = "0.4.0"
clap = "2.23.3"
csv = "0.15.0"
error-chain = "0.11.0"
flexi_logger = "0.6.9"
futures = "0.1.13"
futures-cpupool = "0.1.8"
http-box = "0.1.5"
httparse = "1.2.2"
humansize = "1.0.1"
itertools = "0.7.6"
log = "0.4.1"
mysql_async = "0.12.0"
native-tls = "0.1.2"
ndarray = "0.11.1"
nom = "3.0.0"
revord = "0.0.2"
serde = "1.0.6"
serde_derive = "1.0.6"
serde_json = "1.0.2"
serde_yaml = "0.7.1"
textwrap = "0.9.0"
time = "0.1.37"
tokio-core = "0.1.7"
tokio-io = "0.1.1"
tokio-service = "0.1.0"
tokio-timer = "0.1.1"
tokio-tls = "0.1.2"

@P-E-Meunier
Copy link
Contributor Author

@boxofrox: it's ok, I can reproduce. I believe it's fixed in Carnix 0.6.5.

(also, some of @Mic92's comments are implemented in 0.6.5)

@Mic92 Mic92 mentioned this pull request Feb 3, 2018
8 tasks
@Mic92
Copy link
Member

Mic92 commented Feb 3, 2018

replaced by #34568

@Mic92 Mic92 closed this Feb 3, 2018
@boxofrox
Copy link
Contributor

boxofrox commented Feb 4, 2018

One more comment to note that Carnix 0.6.5 resolves my issue regarding 0.6.4. Many thanks @P-E-Meunier.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants