Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skaware summer 2018 release #45970

Merged
merged 5 commits into from Sep 6, 2018

Conversation

Profpatsch
Copy link
Member

@Profpatsch Profpatsch commented Sep 3, 2018

Motivation for this change

https://www.mail-archive.com/skaware@list.skarnet.org/msg01217.html

Each package in the skaware stack is structured pretty much exactly the same way, so there was tremendous duplication in the way those packages were defined. This removes most of that duplication.

The commits are incremental and are best reviewed one-by-one.

The nsss package is added, most of the stack supports optionally using this library, this is not yet added (if we want to run the tests for nsss there is a circular dependency on s6 to make things harder).

cc other maintainer @pmahoney

Closes #45066.

  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all new binary files (usually in ./result/bin/)

They are normally updated in lockstep, this makes maintenance more
convenient.
Introduce a `skawarePackages.buildPackage` function that contains the
common setup, removing a lot of duplication.
In particular, we require that the build directory has to be empty
after the `fixupPhase`, to make sure every relevant file is moved to
the outputs.

A next step would be to deduplicate the `configureFlags` attributes
and only require a `skawareInputs` field.
name = "${pname}-${version}";

src = fetchgit {
url = "git://git.skarnet.org/${pname}";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a non git protocol alternative (preferable even https)? The git protocol breaks for some of our users because of proxies and has the disadvantage that trust-on-first-use is not secured by https when updating the checksum.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I believe we can switch to https. I didn’t have time to do that yesterday.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

fi
'';

meta = {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The disadvantage of having meta here though is that nix edit breaks: #45948

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe meta.description could stay in the file itself and all other information is overridden here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh, that’s bad. I had a hunch there would be tooling that is too dumb for refactoring meta.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something like

meta  = {
  default = ""
} // userMeta;

should work no? And it also has the advantage that the reader does not have to learn about the new function because the API stays the same.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, after trying it out it looks like I need to have the meta attribute in the package calling the function. I’m not in favor of duplication to support hacks like this. :(

mv doc $doc/share/doc/s6-networking/html
'';
# remove all s6 executables from build directory
rm $(find -name "s6-*" -type f -mindepth 1 -maxdepth 1 -executable)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is cleaning the build directory necessary?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It’s an experiment of mine, being more strict with how build artifacts are handled. The idea is that every file has to be either moved to an output or deleted.
If a future update introduces e.g. an example folder, the maintainer will be alerted that there’s new files that should probably go to an output.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think it is a common problem? Did you found any examples where this was the case for Skaware's software?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don’t know if it’s a common problem, other builds don’t register when new files are added. I strongly suspect it, since many (many!) packages don’t copy documentation, READMEs, or even manpages to their outputs.

Copy link
Member

@Mic92 Mic92 Sep 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not so sure if many people are reading documentation from the nix store except for man/info pages though, which might be the reason, why it not installed by the build system in the first place. I know that debian has a policy that every command must have a man page.

"src/**/*"
"tools/**/*"
"package/**/*"
"config.mak"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do they have be deleted from the source directory?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above.

@GrahamcOfBorg
Copy link

Success on x86_64-darwin (full log)

Attempted: skawarePackages.nsss

Partial log (click to expand)

stripping (with command strip and flags -S) in /nix/store/00z4k71677i224d2rhrr8vck8zi6vysv-nsss-0.0.1.0-lib/lib
patching script interpreter paths in /nix/store/00z4k71677i224d2rhrr8vck8zi6vysv-nsss-0.0.1.0-lib
strip is /nix/store/df6k4mgdjxciy0f637lryp7c9ln7n1m3-cctools-binutils-darwin/bin/strip
patching script interpreter paths in /nix/store/9c8r0wyywk2c9hl0s61v55i9gz0lcdj2-nsss-0.0.1.0-dev
strip is /nix/store/df6k4mgdjxciy0f637lryp7c9ln7n1m3-cctools-binutils-darwin/bin/strip
patching script interpreter paths in /nix/store/2m9i3diqv8kl4ja7n2bd2dgps5jhf03v-nsss-0.0.1.0-doc
strip is /nix/store/df6k4mgdjxciy0f637lryp7c9ln7n1m3-cctools-binutils-darwin/bin/strip
patching script interpreter paths in /nix/store/syc16wifhvl7xbda7z6d38qmg67giivw-nsss-0.0.1.0
Checking for remaining source files
/nix/store/9vf2ajdipzs12vj86ashv98j18f27vk8-nsss-0.0.1.0-bin

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: skawarePackages.nsss

Partial log (click to expand)

shrinking RPATHs of ELF executables and libraries in /nix/store/c703lk32lhxf0s7p38m5b9li3p80434c-nsss-0.0.1.0-doc
strip is /nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/c703lk32lhxf0s7p38m5b9li3p80434c-nsss-0.0.1.0-doc
checking for references to /build in /nix/store/c703lk32lhxf0s7p38m5b9li3p80434c-nsss-0.0.1.0-doc...
shrinking RPATHs of ELF executables and libraries in /nix/store/fjwwx56gygawfal500f32f3rngsfz51r-nsss-0.0.1.0
strip is /nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/fjwwx56gygawfal500f32f3rngsfz51r-nsss-0.0.1.0
checking for references to /build in /nix/store/fjwwx56gygawfal500f32f3rngsfz51r-nsss-0.0.1.0...
Checking for remaining source files
/nix/store/g3vl2kkjqhnxyvcpqb6xpy66gp633vdv-nsss-0.0.1.0-bin

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: skawarePackages.nsss

Partial log (click to expand)

shrinking RPATHs of ELF executables and libraries in /nix/store/vv77zw9dfbv0afwvpkvg87v4rhyg482n-nsss-0.0.1.0-doc
strip is /nix/store/y4ymnvgxygpq05h03kyzbj572zmh6zla-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/vv77zw9dfbv0afwvpkvg87v4rhyg482n-nsss-0.0.1.0-doc
checking for references to /build in /nix/store/vv77zw9dfbv0afwvpkvg87v4rhyg482n-nsss-0.0.1.0-doc...
shrinking RPATHs of ELF executables and libraries in /nix/store/rmiwa073pkccxbdv4rh82m2c4kqj0xcl-nsss-0.0.1.0
strip is /nix/store/y4ymnvgxygpq05h03kyzbj572zmh6zla-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/rmiwa073pkccxbdv4rh82m2c4kqj0xcl-nsss-0.0.1.0
checking for references to /build in /nix/store/rmiwa073pkccxbdv4rh82m2c4kqj0xcl-nsss-0.0.1.0...
Checking for remaining source files
/nix/store/gnnjvzbsfsl346gx6mxmk4hydr9g1d2n-nsss-0.0.1.0-bin

@Profpatsch
Copy link
Member Author

@GrahamcOfBorg build skalibs s6-linux-utils execline s6-portable-utils s6-dns s6-networking s6-rc s6

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: skalibs, s6-linux-utils, execline, s6-portable-utils, s6-dns, s6-networking, s6-rc, s6

Partial log (click to expand)

checking for references to /build in /nix/store/hfcwmf0lvbfb8yi3p88gxyb8rg77ynk1-s6-rc-0.4.1.0...
Checking for remaining source files
/nix/store/jkgdsxm15mnjqx3hi27wsf22655gx9fs-skalibs-2.7.0.0-lib
/nix/store/nz6816pnrb2xcn19ispm38dag7cg91az-s6-linux-utils-2.5.0.0-bin
/nix/store/2l42ipjaxrdsr7v1ijllnasr4980k4q4-execline-2.5.0.1-bin
/nix/store/3hmv82xqrixb2pbz61x618qb45fg28i3-s6-portable-utils-2.2.1.2-bin
/nix/store/6ldkklc6zrap7n4j7ncshc84232yppwj-s6-dns-2.3.0.1-bin
/nix/store/j192jahc7yrjlwzz702zhv42kwdqpkqm-s6-networking-2.3.0.3-bin
/nix/store/620v85h64fm336hznwycpz7yb91v8v3n-s6-rc-0.4.1.0-bin
/nix/store/r53sfpdbf1jbz9n1mzsgkn502z7dnicg-s6-2.7.2.0

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: skalibs, s6-linux-utils, execline, s6-portable-utils, s6-dns, s6-networking, s6-rc, s6

Partial log (click to expand)

checking for references to /build in /nix/store/nrx6mnh3ndl1s2vyx95gnmaylmw6hg1s-s6-rc-0.4.1.0...
Checking for remaining source files
/nix/store/v0sy6g2v5jlid7fdhmxpx0wfg34a56f0-skalibs-2.7.0.0-lib
/nix/store/jsbds5d1rknvl2y9gc5qw1kkpszqa1xd-s6-linux-utils-2.5.0.0-bin
/nix/store/vl8n9gh9d5m1fazw9sb3i9hx0w6c8m3i-execline-2.5.0.1-bin
/nix/store/ag35kdzk5chy26hjzr8yw4giyykazm6n-s6-portable-utils-2.2.1.2-bin
/nix/store/z8fq3rycq58w08mx55vx8w785fky0mab-s6-dns-2.3.0.1-bin
/nix/store/aivv3lqcjvl4wd9wv5rq8qcnpma6rcdc-s6-networking-2.3.0.3-bin
/nix/store/61rng91g0qnlh79dfczacblchqj4xqg8-s6-rc-0.4.1.0-bin
/nix/store/5csn6bjinaykcyjmy9lx92nh9jz37n3h-s6-2.7.2.0

@GrahamcOfBorg
Copy link

Success on x86_64-darwin (full log)

Attempted: skalibs, execline, s6-portable-utils, s6-dns, s6-networking, s6

The following builds were skipped because they don't evaluate on x86_64-darwin: s6-linux-utils, s6-rc

Partial log (click to expand)

patching script interpreter paths in /nix/store/8n560a7f73wl0py4am68cxx57ybvdd3r-s6-networking-2.3.0.3-doc
strip is /nix/store/df6k4mgdjxciy0f637lryp7c9ln7n1m3-cctools-binutils-darwin/bin/strip
patching script interpreter paths in /nix/store/9ysrvdiask981q3xnqg9bc6j9siwqw8c-s6-networking-2.3.0.3
Checking for remaining source files
/nix/store/rkhnv7hdqr47phqpvnlnwy46lz01nhmn-skalibs-2.7.0.0-lib
/nix/store/rsq9nq5jn2kg82ki2k0wrlld5j2mwl7w-execline-2.5.0.1-bin
/nix/store/mblxwl24cs8v86wpz7r90v1lxf050qy9-s6-portable-utils-2.2.1.2-bin
/nix/store/df9hcfs848gjvx5wkdyvgqzvs8s4sdil-s6-dns-2.3.0.1-bin
/nix/store/wm961fpvkb5bhzffa3np026836206hw8-s6-networking-2.3.0.3-bin
/nix/store/122vflh4mckzwv8w16hbvvgggl38653f-s6-2.7.2.0

It should be more performant this way.
@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: skawarePackages.nsss

Partial log (click to expand)

checking for references to /build in /nix/store/s48a9ydsv23kam91kf8dybb20swdd3vs-nsss-0.0.1.0-dev...
shrinking RPATHs of ELF executables and libraries in /nix/store/i78nczkavrzac161d0ja7ifbww1qgm5q-nsss-0.0.1.0-doc
strip is /nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/i78nczkavrzac161d0ja7ifbww1qgm5q-nsss-0.0.1.0-doc
checking for references to /build in /nix/store/i78nczkavrzac161d0ja7ifbww1qgm5q-nsss-0.0.1.0-doc...
shrinking RPATHs of ELF executables and libraries in /nix/store/lj7cyzswbrcv2z6ji0xrv985nj2rhmli-nsss-0.0.1.0
strip is /nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/lj7cyzswbrcv2z6ji0xrv985nj2rhmli-nsss-0.0.1.0
checking for references to /build in /nix/store/lj7cyzswbrcv2z6ji0xrv985nj2rhmli-nsss-0.0.1.0...
Checking for remaining source files

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: skawarePackages.nsss

Partial log (click to expand)

shrinking RPATHs of ELF executables and libraries in /nix/store/m57dyj9gvjy40n1lmkav2ixk0xmbrgi1-nsss-0.0.1.0-doc
strip is /nix/store/y4ymnvgxygpq05h03kyzbj572zmh6zla-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/m57dyj9gvjy40n1lmkav2ixk0xmbrgi1-nsss-0.0.1.0-doc
checking for references to /build in /nix/store/m57dyj9gvjy40n1lmkav2ixk0xmbrgi1-nsss-0.0.1.0-doc...
shrinking RPATHs of ELF executables and libraries in /nix/store/8qdjk2q3xnv8krv5533r5r261ngg9ksk-nsss-0.0.1.0
strip is /nix/store/y4ymnvgxygpq05h03kyzbj572zmh6zla-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/8qdjk2q3xnv8krv5533r5r261ngg9ksk-nsss-0.0.1.0
checking for references to /build in /nix/store/8qdjk2q3xnv8krv5533r5r261ngg9ksk-nsss-0.0.1.0...
Checking for remaining source files
/nix/store/wfwr9xs6hyibphj0g4rbpx6ckz92lvz9-nsss-0.0.1.0-bin

@GrahamcOfBorg
Copy link

Success on x86_64-darwin (full log)

Attempted: skawarePackages.nsss

Partial log (click to expand)

stripping (with command strip and flags -S) in /nix/store/hrq7h8hbrvsh9rhqavlasj8fvn0arkif-nsss-0.0.1.0-lib/lib
patching script interpreter paths in /nix/store/hrq7h8hbrvsh9rhqavlasj8fvn0arkif-nsss-0.0.1.0-lib
strip is /nix/store/df6k4mgdjxciy0f637lryp7c9ln7n1m3-cctools-binutils-darwin/bin/strip
patching script interpreter paths in /nix/store/7flnicpy16fnpy8a1rhv8101kf1izjdk-nsss-0.0.1.0-dev
strip is /nix/store/df6k4mgdjxciy0f637lryp7c9ln7n1m3-cctools-binutils-darwin/bin/strip
patching script interpreter paths in /nix/store/ni594i56bf6dnfh1bm9ap4h8d37kxvvd-nsss-0.0.1.0-doc
strip is /nix/store/df6k4mgdjxciy0f637lryp7c9ln7n1m3-cctools-binutils-darwin/bin/strip
patching script interpreter paths in /nix/store/pnjbskn5yh2a4jfdjkim2xkr1ac1kvd9-nsss-0.0.1.0
Checking for remaining source files
/nix/store/pgb32mrm0zv2v65nbnx5j4rz1a4dl0zg-nsss-0.0.1.0-bin

@Profpatsch
Copy link
Member Author

Build with new https hashes:
@GrahamcOfBorg build skalibs s6-linux-utils execline s6-portable-utils s6-dns s6-networking s6-rc s6

This PR is ready from my side.

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: skalibs, s6-linux-utils, execline, s6-portable-utils, s6-dns, s6-networking, s6-rc, s6

Partial log (click to expand)

checking for references to /build in /nix/store/ximw8xb385h53rfmg8ikqyb7bih6afr2-s6-rc-0.4.1.0...
Checking for remaining source files
/nix/store/v700gh0xrizvk0r4pxfqnlx69css802b-skalibs-2.7.0.0-lib
/nix/store/hjqjhmvfpdjg4qapii0m2wj5kam0faqz-s6-linux-utils-2.5.0.0-bin
/nix/store/w7p0f2mbzf648zbf8z0njmdgigpblhgr-execline-2.5.0.1-bin
/nix/store/f9bgi4s3bd4nxd83gvkib2zmxj0wn1x4-s6-portable-utils-2.2.1.2-bin
/nix/store/mq4hm5al1db4jbi9cf3dzk0fdnmvsv9g-s6-dns-2.3.0.1-bin
/nix/store/cr4nlv80dk4hlmac3il01vg3y8pihy6l-s6-networking-2.3.0.3-bin
/nix/store/gz37bi4adq0x3gm1yxlr6gq5i79ny4vz-s6-rc-0.4.1.0-bin
/nix/store/4gczxy1v63ypm26sifg4y44ypzj8mfa2-s6-2.7.2.0

@GrahamcOfBorg
Copy link

Success on x86_64-darwin (full log)

Attempted: skalibs, execline, s6-portable-utils, s6-dns, s6-networking, s6

The following builds were skipped because they don't evaluate on x86_64-darwin: s6-linux-utils, s6-rc

Partial log (click to expand)

patching script interpreter paths in /nix/store/s167n75ch702plb3nws0rppv2z163i4y-s6-portable-utils-2.2.1.2-doc
strip is /nix/store/df6k4mgdjxciy0f637lryp7c9ln7n1m3-cctools-binutils-darwin/bin/strip
patching script interpreter paths in /nix/store/ba0xwb39ckxwsci08vmwiwisraz84baf-s6-portable-utils-2.2.1.2
Checking for remaining source files
/nix/store/s8bf2ji5w2rkixp7mbggkz9bnfmr2frh-skalibs-2.7.0.0-lib
/nix/store/3grv4gn2p6mbfp0y5lxz5zh9wv44nwq4-execline-2.5.0.1-bin
/nix/store/kj3xmqh3h4i7r4yz71q11aiylib5hyx4-s6-portable-utils-2.2.1.2-bin
/nix/store/z78r5aizi13842i8nzzl4bjhrcxqlqd7-s6-dns-2.3.0.1-bin
/nix/store/jjqjc960xga8iac17p20yg1l9v5rwhh4-s6-networking-2.3.0.3-bin
/nix/store/qlfivgf0lsi4dlwaq0ks2k9m4yy1rwqn-s6-2.7.2.0

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: skalibs, s6-linux-utils, execline, s6-portable-utils, s6-dns, s6-networking, s6-rc, s6

Partial log (click to expand)

checking for references to /build in /nix/store/md4vash46cyknimvbq06dksxnm5xhcar-s6-rc-0.4.1.0...
Checking for remaining source files
/nix/store/zwcj4x3ang9609qk5v8p941pama6x5wa-skalibs-2.7.0.0-lib
/nix/store/cykyfjnwwi491az0pc0k61wl2jblaffr-s6-linux-utils-2.5.0.0-bin
/nix/store/2f662r9x1412xp4z2qbgab7wxxr7z2sp-execline-2.5.0.1-bin
/nix/store/bhyhc9l6dywcrwnckqp47mlac4zm5na8-s6-portable-utils-2.2.1.2-bin
/nix/store/dp4lhsnb45z5738c571kv62xx9div796-s6-dns-2.3.0.1-bin
/nix/store/d35riv5igywmmmyjqcbirqmm12fw139f-s6-networking-2.3.0.3-bin
/nix/store/la76mqafcnvz06nwk7plpqbkaqf5y4l9-s6-rc-0.4.1.0-bin
/nix/store/wyykiq3hy1mw30fggvbps5m85q5b2yjh-s6-2.7.2.0

@Profpatsch Profpatsch merged commit 4616ef1 into NixOS:master Sep 6, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants