Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

libxml2: add static output #41915

Merged
merged 1 commit into from Jun 13, 2018
Merged

libxml2: add static output #41915

merged 1 commit into from Jun 13, 2018

Conversation

nlewo
Copy link
Member

@nlewo nlewo commented Jun 13, 2018

I don't know if this has to be conditional (such as pkgs.zlib that have a boolean static argument) or not (such as pkgs.glibc).

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

@nlewo
Copy link
Member Author

nlewo commented Jun 13, 2018

@GrahamcOfBorg build libxml2.static

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: libxml2.static

Partial log (click to expand)

strip is /nix/store/21ymadblbmsbb2bk4q7gl4kjasp8zmgd-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/m69hvqzz4jqrqa2d5xawrl45mm73ph38-libxml2-2.9.8-py/lib
patching script interpreter paths in /nix/store/m69hvqzz4jqrqa2d5xawrl45mm73ph38-libxml2-2.9.8-py
checking for references to /build in /nix/store/m69hvqzz4jqrqa2d5xawrl45mm73ph38-libxml2-2.9.8-py...
Moving /nix/store/ydq664x5qj6ivg8lbnd0zw60ay4sg4rg-libxml2-2.9.8-bin/bin/xml2-config to /nix/store/vmw08nhs1y5g7612dfyd7wcr9j17cvlk-libxml2-2.9.8-dev/bin/xml2-config
rmdir: failed to remove '/nix/store/ydq664x5qj6ivg8lbnd0zw60ay4sg4rg-libxml2-2.9.8-bin/bin': Directory not empty
Moving /nix/store/3ma1b5pbiyvrdl11rdi35sa65716yr9f-libxml2-2.9.8/lib/xml2Conf.sh to /nix/store/vmw08nhs1y5g7612dfyd7wcr9j17cvlk-libxml2-2.9.8-dev/lib/xml2Conf.sh
rmdir: failed to remove '/nix/store/3ma1b5pbiyvrdl11rdi35sa65716yr9f-libxml2-2.9.8/lib': Directory not empty
Moving /nix/store/g9nv776rfqsb30yjsc1zagncqq43dfpl-libxml2-2.9.8-man/share/man/man1 to /nix/store/ydq664x5qj6ivg8lbnd0zw60ay4sg4rg-libxml2-2.9.8-bin/share/man/man1
rmdir: failed to remove '/nix/store/g9nv776rfqsb30yjsc1zagncqq43dfpl-libxml2-2.9.8-man/share/man': Directory not empty

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: libxml2.static

Partial log (click to expand)

stripping (with command strip and flags -S) in /nix/store/n73h3w82qs8igahzv74z77nxgpa8jdz6-libxml2-2.9.8-py/lib
patching script interpreter paths in /nix/store/n73h3w82qs8igahzv74z77nxgpa8jdz6-libxml2-2.9.8-py
checking for references to /build in /nix/store/n73h3w82qs8igahzv74z77nxgpa8jdz6-libxml2-2.9.8-py...
Moving /nix/store/m6cl86dhdf0hmz4d8a0jsv3nkdlfnzqs-libxml2-2.9.8-bin/bin/xml2-config to /nix/store/iqr7nzrckib9mzbpqa8cmxvp1dplawhy-libxml2-2.9.8-dev/bin/xml2-config
rmdir: failed to remove '/nix/store/m6cl86dhdf0hmz4d8a0jsv3nkdlfnzqs-libxml2-2.9.8-bin/bin': Directory not empty
Moving /nix/store/zlj8hcqsyc6r7i83d2fv5lb1sf9inqjr-libxml2-2.9.8/lib/xml2Conf.sh to /nix/store/iqr7nzrckib9mzbpqa8cmxvp1dplawhy-libxml2-2.9.8-dev/lib/xml2Conf.sh
rmdir: failed to remove '/nix/store/zlj8hcqsyc6r7i83d2fv5lb1sf9inqjr-libxml2-2.9.8/lib': Directory not empty
Moving /nix/store/qkcxm78n652r48sq6x2nla8j8mf3347i-libxml2-2.9.8-man/share/man/man1 to /nix/store/m6cl86dhdf0hmz4d8a0jsv3nkdlfnzqs-libxml2-2.9.8-bin/share/man/man1
rmdir: failed to remove '/nix/store/qkcxm78n652r48sq6x2nla8j8mf3347i-libxml2-2.9.8-man/share/man': Directory not empty
/nix/store/vx03jl3sqqy145jgwvy61ziq2jvfrwbf-libxml2-2.9.8-static

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: libxml2

Partial log (click to expand)

/nix/store/m6cl86dhdf0hmz4d8a0jsv3nkdlfnzqs-libxml2-2.9.8-bin

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: libxml2

Partial log (click to expand)

strip is /nix/store/21ymadblbmsbb2bk4q7gl4kjasp8zmgd-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/m69hvqzz4jqrqa2d5xawrl45mm73ph38-libxml2-2.9.8-py/lib
patching script interpreter paths in /nix/store/m69hvqzz4jqrqa2d5xawrl45mm73ph38-libxml2-2.9.8-py
checking for references to /build in /nix/store/m69hvqzz4jqrqa2d5xawrl45mm73ph38-libxml2-2.9.8-py...
Moving /nix/store/ydq664x5qj6ivg8lbnd0zw60ay4sg4rg-libxml2-2.9.8-bin/bin/xml2-config to /nix/store/vmw08nhs1y5g7612dfyd7wcr9j17cvlk-libxml2-2.9.8-dev/bin/xml2-config
rmdir: failed to remove '/nix/store/ydq664x5qj6ivg8lbnd0zw60ay4sg4rg-libxml2-2.9.8-bin/bin': Directory not empty
Moving /nix/store/3ma1b5pbiyvrdl11rdi35sa65716yr9f-libxml2-2.9.8/lib/xml2Conf.sh to /nix/store/vmw08nhs1y5g7612dfyd7wcr9j17cvlk-libxml2-2.9.8-dev/lib/xml2Conf.sh
rmdir: failed to remove '/nix/store/3ma1b5pbiyvrdl11rdi35sa65716yr9f-libxml2-2.9.8/lib': Directory not empty
Moving /nix/store/g9nv776rfqsb30yjsc1zagncqq43dfpl-libxml2-2.9.8-man/share/man/man1 to /nix/store/ydq664x5qj6ivg8lbnd0zw60ay4sg4rg-libxml2-2.9.8-bin/share/man/man1
rmdir: failed to remove '/nix/store/g9nv776rfqsb30yjsc1zagncqq43dfpl-libxml2-2.9.8-man/share/man': Directory not empty

@xeji
Copy link
Contributor

xeji commented Jun 13, 2018

Does this build on Darwin?
@GrahamcOfBorg build libxml2.static

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: libxml2.static

Partial log (click to expand)

/nix/store/qj7ywh5r0168cwcpk4nvh4727zdcpj1c-libxml2-2.9.8-static

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: libxml2.static

Partial log (click to expand)

/nix/store/vx03jl3sqqy145jgwvy61ziq2jvfrwbf-libxml2-2.9.8-static

@GrahamcOfBorg
Copy link

Failure on x86_64-darwin (full log)

Attempted: libxml2.static

Partial log (click to expand)

cannot build derivation '/nix/store/gc1vc4r55ivlmqp5ivzpnv7rg1fcswzi-libtool-2.4.6.drv': 4 dependencies couldn't be built
cannot build derivation '/nix/store/fkr6clwqyvhhk6i6cflj4vp8ypvd9j6h-hook.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/ixs4lxyzddx2kj2wl0hlbdp4hwl52g9v-flex-2.6.4.drv': 6 dependencies couldn't be built
cannot build derivation '/nix/store/i8mq3y9lg4099pkqbq8jiq0d6gp3lh08-bootstrap_cmds-dev-tools-7.0.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/jzgzacb1p4awjk49sfdxwkkarvd7d37g-xnu-osx-10.11.6.drv': 8 dependencies couldn't be built
cannot build derivation '/nix/store/29qmchf2p6k4ai8gfiznfw1l4zrvlgya-IOKit-osx-10.11.6.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/jizrjpqa425s29khxq40j55a2k55m17v-configd-osx-10.8.5.drv': 8 dependencies couldn't be built
cannot build derivation '/nix/store/jllm5difcidmgjfvvinhwsj6s30qlip7-python-2.7.15.drv': 9 dependencies couldn't be built
cannot build derivation '/nix/store/vkazax5spgmy1zdf21dynpkp01mjcbb6-libxml2-2.9.8.drv': 3 dependencies couldn't be built
error: build of '/nix/store/vkazax5spgmy1zdf21dynpkp01mjcbb6-libxml2-2.9.8.drv' failed

@nlewo
Copy link
Member Author

nlewo commented Jun 13, 2018

The failure on Darwin is due to a timeout on a dependency.

@xeji
Copy link
Contributor

xeji commented Jun 13, 2018

This causes a large number of rebuilds, but it does not make any functional changes, so I think it's ok for master.

@xeji xeji merged commit ff842fa into NixOS:master Jun 13, 2018
@matthewbauer
Copy link
Member

I think we need to standardize how to handle static binaries. I'd prefer to make these optional with a flag like "enableStatic". Doing this for every package that can build statically doesn't make sense. Multiple outputs can still be used - it's just building statically every time seems like a huge added cost in aggregate. See: #39580

orivej pushed a commit that referenced this pull request Jun 13, 2018
@nlewo @orivej
libxml2: add static output (#41915)
819f47b
@xeji
Copy link
Contributor

xeji commented Jun 13, 2018

I agree that a standardized flag is a good idea. But I don't see a significant added cost in this case. It's a library, and all this does is put the already compiled .o's into an additional .a archive.

Building entire applications statically is a different story... Maybe we should distinguish between building libraries (always with static output if supported) and applications (controlled by a global flag)

@orivej
Copy link
Contributor

orivej commented Jun 13, 2018

This causes a large number of rebuilds, but it does not make any functional changes, so I think it's ok for master.

The purpose of staging is to accumulate mass rebuilds, not to delay risky changes. Otherwise new pull requests can not be tested while master is being rebuilt, and it takes longer to notice any breakages on https://hydra.nixos.org/jobset/nixpkgs/trunk. I have reverted this on master in ea69fba and applied to staging in 819f47b.

@matthewbauer
Copy link
Member

Shouldn't libxml2 install the .a files on its own? Anyway having a standardized ".a" output in https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/setup-hooks/multiple-outputs.sh would be really useful for these cases anyway (we never want to put dynamic & static libraries side by side otherwise we will have huge closures).

@xeji
Copy link
Contributor

xeji commented Jun 13, 2018

Shouldn't libxml2 install the .a files on its own.

It apparently does that only when configured with --enable-static, which is what this PR does.

Anyway having a standardized ".a" output in https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/setup-hooks/multiple-outputs.sh would be really useful for these cases anyway

Sounds good.

@orivej
Copy link
Contributor

orivej commented Jun 13, 2018

Shouldn't libxml2 install the .a files on its own?

It does, now there is lib/libxml2.a in the default and the static outputs. The proper fix is to use moveToOutput in the postFixup phase and delete postInstall, but I'm not sure if it should be fixed or just reverted because it does not seem useful in Nixpkgs. @nlewo, could you provide motivation for this change?

orivej added a commit that referenced this pull request Jun 13, 2018
@orivej
Revert "libxml2: add static output (#41915)"
f5c8402 
This reverts commit 819f47b on staging to
prevent unneeded rebuild.

That change leaves the static library in two outputs, "out" and "static", and
has to be reimplemented and motivated.
@orivej
Copy link
Contributor

orivej commented Jun 13, 2018 •
edited

I have reverted this on staging too in f5c8402 because it has to be changed anyway. You can submit a new PR for staging that moves the static library in postFixup if you think that this belongs in Nixpkgs. (Alternatively you can make the static output optional and submit a PR for master.)

@nlewo
Copy link
Member Author

nlewo commented Jun 14, 2018

@orivej I need it to produce a static Go binary which also uses glibc.static and zlibStatic. The main objective is to be able to copy it and run it on a non "Nix system".

I will open a new PR, based on staging, with a static output and requested changes.

Thanks for all of your comments!

@nlewo
Copy link
Member Author

nlewo commented Jun 14, 2018

Or wait for #41935 be merged :/

@orivej
Copy link
Contributor

orivej commented Jun 15, 2018

glibc.static

Doesn't it require a compatible version of a dynamic glibc on the target system, as in https://tschottdorf.github.io/golang-static-linking-bug ? (But I suppose you could link with dynamic glibc yet static libxml2 and zlib.)

The main objective is to be able to copy it and run it on a non "Nix system".

You may consider bundling your application with all dynamic dependencies, as in https://github.com/orivej/ldcp/ .

@nlewo
Copy link
Member Author

nlewo commented Jun 16, 2018

@orivej I no longer have such binaries to get more details but I don't think it required a global glibc since I could run the produced static binary in a NixOS test VM (sandboxed).

For my current usecase, I'd prefer to use static libraries because the project I'm compiling is doing this upstream, on Debian and Fedora distros. so I just want to do the same on NixOS.
But thanks for ldcp, this could be useful in different contexts:/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
10.rebuild-darwin: 501+ 10.rebuild-darwin-stdenv 10.rebuild-linux: 501+
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@nlewo @GrahamcOfBorg @xeji @matthewbauer @orivej