Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stdenv: enable colors #42932

Closed
wants to merge 1 commit into from
Closed

Conversation

infinisil
Copy link
Member

@infinisil infinisil commented Jul 3, 2018

Since the builds are running in the sandbox, there can't be a way for a
tool like cmake to correctly detect whether the output is a tty or not,
so colors will be disabled in all cases, even when you build with
nix-build and see the lines scrolling by in your terminal.

To have colors we therefore need to force it to use them with the
dedicated env var. Note that simply running CLICOLOR_FORCE=1 cmake ...
won't do it, we need to actually export it, so subprocesses can see
it.

I used aws-sdk-cpp to test it on (note that if you run cmake from this PR, it will first take a while to compile cmake itself, which will still be uncolored).

Motivation for this change

Colors can be very useful to quickly spot errors/warnings in a build. Here is a recording of a comparison of a build with and without colors: https://asciinema.org/a/WNDtvyeRH5cDmq0EqiZrpN5M0

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

@grahamc
Copy link
Member

grahamc commented Jul 3, 2018

@GrahamcOfBorg build aws-sdk-cpp

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: cmake

Partial log (click to expand)

/nix/store/dh7dl8m6bs48zdmzag8p72z40737nkqk-cmake-3.11.2

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: cmake

Partial log (click to expand)

shrinking /nix/store/yyiwyn15rh34nrs1m454i6ix4sm31x8q-cmake-3.11.2/bin/cmake
shrinking /nix/store/yyiwyn15rh34nrs1m454i6ix4sm31x8q-cmake-3.11.2/bin/ctest
shrinking /nix/store/yyiwyn15rh34nrs1m454i6ix4sm31x8q-cmake-3.11.2/bin/cpack
strip is /nix/store/4qvrxzxa535y8304mk195x50b6p9607d-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/yyiwyn15rh34nrs1m454i6ix4sm31x8q-cmake-3.11.2/bin
patching script interpreter paths in /nix/store/yyiwyn15rh34nrs1m454i6ix4sm31x8q-cmake-3.11.2
/nix/store/yyiwyn15rh34nrs1m454i6ix4sm31x8q-cmake-3.11.2/share/cmake-3.11/Modules/Squish4RunTestCase.sh: interpreter directive changed from "/bin/sh" to "/nix/store/8zkg9ac4s4alzyf4a8kfrig1j73z66dw-bash-4.4-p23/bin/sh"
/nix/store/yyiwyn15rh34nrs1m454i6ix4sm31x8q-cmake-3.11.2/share/cmake-3.11/Modules/CPack.STGZ_Header.sh.in: interpreter directive changed from "/bin/sh" to "/nix/store/8zkg9ac4s4alzyf4a8kfrig1j73z66dw-bash-4.4-p23/bin/sh"
/nix/store/yyiwyn15rh34nrs1m454i6ix4sm31x8q-cmake-3.11.2/share/cmake-3.11/Modules/SquishRunTestCase.sh: interpreter directive changed from "/bin/sh" to "/nix/store/8zkg9ac4s4alzyf4a8kfrig1j73z66dw-bash-4.4-p23/bin/sh"
checking for references to /build in /nix/store/yyiwyn15rh34nrs1m454i6ix4sm31x8q-cmake-3.11.2...

@infinisil infinisil force-pushed the cmake-colors branch 2 times, most recently from 8ab86d6 to 58742fe Compare July 3, 2018 19:46
@infinisil infinisil requested a review from FRidh as a code owner July 3, 2018 19:46
@GrahamcOfBorg
Copy link

Failure on x86_64-linux (full log)

Attempted: aws-sdk-cpp

Partial log (click to expand)

[ 37%] Building CXX object aws-cpp-sdk-es/CMakeFiles/aws-cpp-sdk-es.dir/source/model/AddTagsRequest.cpp.o
[ 37%] Building CXX object aws-cpp-sdk-elasticmapreduce/CMakeFiles/aws-cpp-sdk-elasticmapreduce.dir/source/model/SpotProvisioningSpecification.cpp.o
[ 37%] Building CXX object aws-cpp-sdk-email/CMakeFiles/aws-cpp-sdk-email.dir/source/model/DeleteIdentityPolicyResult.cpp.o
[ 37%] Building CXX object aws-cpp-sdk-ec2/CMakeFiles/aws-cpp-sdk-ec2.dir/source/model/DescribeSnapshotAttributeResponse.cpp.o
[ 37%] Building CXX object aws-cpp-sdk-elasticmapreduce/CMakeFiles/aws-cpp-sdk-elasticmapreduce.dir/source/model/SpotProvisioningTimeoutAction.cpp.o
[ 37%] Building CXX object aws-cpp-sdk-email/CMakeFiles/aws-cpp-sdk-email.dir/source/model/DeleteIdentityRequest.cpp.o
[ 37%] Building CXX object aws-cpp-sdk-es/CMakeFiles/aws-cpp-sdk-es.dir/source/model/AdditionalLimit.cpp.o
[ 37%] Building CXX object aws-cpp-sdk-ec2/CMakeFiles/aws-cpp-sdk-ec2.dir/source/model/DescribeSnapshotsRequest.cpp.o
building of '/nix/store/hmmydpyz91vkrkywjz3kan1bly8fd7qv-aws-sdk-cpp-1.4.70.drv' timed out after 1800 seconds
error: build of '/nix/store/hmmydpyz91vkrkywjz3kan1bly8fd7qv-aws-sdk-cpp-1.4.70.drv' failed

@infinisil infinisil changed the base branch from master to staging July 3, 2018 19:46
@@ -93,6 +93,8 @@ fi

addEnvHooks "$targetOffset" addCMakeParams

export CLICOLOR_FORCE=1
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it makes sense to set this value here, because sometimes packages (e.g. the cmake one itself) use dontUseCmakeConfigure, but still use cmake with custom flags to configure it. These will have colored output too by exporting it here.

@orivej
Copy link
Contributor

orivej commented Jul 3, 2018

CLICOLOR_FORCE is not specific to cmake and is meant to affect other programs too, so if we export it we should do this globally (in the stdenv). However, both ways will be visible in nix-shell, which may be as undesirable as hardening was before #28029.

Note that simply running CLICOLOR_FORCE=1 cmake ... won't do it, we need to actually export it, so subprocesses can see it.

export var=val; program and var=val program have exactly the same effect on the program. The only difference is that export also sets the variable for the current shell and for future programs.

@GrahamcOfBorg
Copy link

Failure on aarch64-linux (full log)

Attempted: aws-sdk-cpp

Partial log (click to expand)

[ 66%] Building CXX object aws-cpp-sdk-ec2/CMakeFiles/aws-cpp-sdk-ec2.dir/source/model/AvailableCapacity.cpp.o
[ 66%] Building CXX object aws-cpp-sdk-gamelift/CMakeFiles/aws-cpp-sdk-gamelift.dir/source/model/PlayerSession.cpp.o
[ 66%] Building CXX object aws-cpp-sdk-sagemaker/CMakeFiles/aws-cpp-sdk-sagemaker.dir/source/model/RecordWrapper.cpp.o
[ 66%] Building CXX object aws-cpp-sdk-rekognition/CMakeFiles/aws-cpp-sdk-rekognition.dir/source/model/S3Object.cpp.o
[ 66%] Building CXX object aws-cpp-sdk-storagegateway/CMakeFiles/aws-cpp-sdk-storagegateway.dir/source/model/DeviceiSCSIAttributes.cpp.o
[ 66%] Building CXX object aws-cpp-sdk-s3/CMakeFiles/aws-cpp-sdk-s3.dir/source/model/GetBucketTaggingRequest.cpp.o
[ 66%] Building CXX object aws-cpp-sdk-waf-regional/CMakeFiles/aws-cpp-sdk-waf-regional.dir/source/model/CreateSqlInjectionMatchSetRequest.cpp.o
[ 66%] Building CXX object aws-cpp-sdk-servicecatalog/CMakeFiles/aws-cpp-sdk-servicecatalog.dir/source/model/DescribeProductViewResult.cpp.o
building of '/nix/store/00aq1vs45as81sf7z57pzscvrnlkrdnj-aws-sdk-cpp-1.4.70.drv' timed out after 3600 seconds
error: build of '/nix/store/00aq1vs45as81sf7z57pzscvrnlkrdnj-aws-sdk-cpp-1.4.70.drv' failed

@infinisil
Copy link
Member Author

However, both ways will be visible in nix-shell, which may be as undesirable as hardening was before #28029.

I fail to get the meaning out of this sentence.

The only difference is that export also sets the variable for the current shell and for future programs.

Learned something new :D

I'll put the export in stdenv then

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: cmake

Partial log (click to expand)

shrinking /nix/store/h68pnwhkxc3jbr6d1ylgyydnqc97wx4h-cmake-3.11.2/bin/cmake
shrinking /nix/store/h68pnwhkxc3jbr6d1ylgyydnqc97wx4h-cmake-3.11.2/bin/ctest
shrinking /nix/store/h68pnwhkxc3jbr6d1ylgyydnqc97wx4h-cmake-3.11.2/bin/cpack
strip is /nix/store/90vmpr41dzsx350k5argycaf693hnl1l-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/h68pnwhkxc3jbr6d1ylgyydnqc97wx4h-cmake-3.11.2/bin
patching script interpreter paths in /nix/store/h68pnwhkxc3jbr6d1ylgyydnqc97wx4h-cmake-3.11.2
/nix/store/h68pnwhkxc3jbr6d1ylgyydnqc97wx4h-cmake-3.11.2/share/cmake-3.11/Modules/Squish4RunTestCase.sh: interpreter directive changed from "/bin/sh" to "/nix/store/z9qg7lpkrajdnlgw5pxp9yx984prl1fi-bash-4.4-p23/bin/sh"
/nix/store/h68pnwhkxc3jbr6d1ylgyydnqc97wx4h-cmake-3.11.2/share/cmake-3.11/Modules/CPack.STGZ_Header.sh.in: interpreter directive changed from "/bin/sh" to "/nix/store/z9qg7lpkrajdnlgw5pxp9yx984prl1fi-bash-4.4-p23/bin/sh"
/nix/store/h68pnwhkxc3jbr6d1ylgyydnqc97wx4h-cmake-3.11.2/share/cmake-3.11/Modules/SquishRunTestCase.sh: interpreter directive changed from "/bin/sh" to "/nix/store/z9qg7lpkrajdnlgw5pxp9yx984prl1fi-bash-4.4-p23/bin/sh"
checking for references to /build in /nix/store/h68pnwhkxc3jbr6d1ylgyydnqc97wx4h-cmake-3.11.2...

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: cmake

Partial log (click to expand)

shrinking /nix/store/flxx347lzhifgi448bif4irimfwqqmxl-cmake-3.11.2/bin/cpack
shrinking /nix/store/flxx347lzhifgi448bif4irimfwqqmxl-cmake-3.11.2/bin/ctest
shrinking /nix/store/flxx347lzhifgi448bif4irimfwqqmxl-cmake-3.11.2/bin/cmake
strip is /nix/store/7iyn7gn33i7xxjgmwf25k20246y6nd9d-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/flxx347lzhifgi448bif4irimfwqqmxl-cmake-3.11.2/bin
patching script interpreter paths in /nix/store/flxx347lzhifgi448bif4irimfwqqmxl-cmake-3.11.2
/nix/store/flxx347lzhifgi448bif4irimfwqqmxl-cmake-3.11.2/share/cmake-3.11/Modules/CPack.STGZ_Header.sh.in: interpreter directive changed from "/bin/sh" to "/nix/store/3swzdhnphnbxjcyx2bkk1gh9k8gi3kl6-bash-4.4-p23/bin/sh"
/nix/store/flxx347lzhifgi448bif4irimfwqqmxl-cmake-3.11.2/share/cmake-3.11/Modules/Squish4RunTestCase.sh: interpreter directive changed from "/bin/sh" to "/nix/store/3swzdhnphnbxjcyx2bkk1gh9k8gi3kl6-bash-4.4-p23/bin/sh"
/nix/store/flxx347lzhifgi448bif4irimfwqqmxl-cmake-3.11.2/share/cmake-3.11/Modules/SquishRunTestCase.sh: interpreter directive changed from "/bin/sh" to "/nix/store/3swzdhnphnbxjcyx2bkk1gh9k8gi3kl6-bash-4.4-p23/bin/sh"
checking for references to /build in /nix/store/flxx347lzhifgi448bif4irimfwqqmxl-cmake-3.11.2...

Since the builds are running in the sandbox, there can't be a way for a
tool like cmake to correctly detect whether the output is a tty or not,
so colors will be disabled in all cases, even when you build with
`nix-build` and see the lines scrolling by in your terminal.

To have colors we therefore need to force it to use them with the
dedicated CLICOLOR_FORCE env var for supported programs. This is somewhat of a
standard, see https://bixense.com/clicolors/
@infinisil infinisil changed the title cmake: enable colors stdenv: enable colors Jul 3, 2018
@@ -203,6 +203,8 @@ rec {
++ optional (elem "host" configurePlatforms) "--host=${stdenv.hostPlatform.config}"
++ optional (elem "target" configurePlatforms) "--target=${stdenv.targetPlatform.config}";

CLICOLOR_FORCE = 1;

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stdenv people: Please let me know if there's a better place to put it than here. It doesn't feel "right" here

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Put it in stdenv/setup.sh?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could probably put it in impureEnvVars. At least, there are definitely times when you may not want colors, for instance in Eshell.

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: stdenv

Partial log (click to expand)

shrinking /nix/store/m7rga8yvy8kzw231qwggly2sbkvqkllc-findutils-4.6.0/libexec/code
shrinking /nix/store/m7rga8yvy8kzw231qwggly2sbkvqkllc-findutils-4.6.0/libexec/bigram
gzipping man pages under /nix/store/m7rga8yvy8kzw231qwggly2sbkvqkllc-findutils-4.6.0/share/man/
strip is /nix/store/d1prcspbh2qsviipvnaxizcj8l3g7fpw-bootstrap-tools/bin/strip
stripping (with command strip and flags -S) in /nix/store/m7rga8yvy8kzw231qwggly2sbkvqkllc-findutils-4.6.0/libexec  /nix/store/m7rga8yvy8kzw231qwggly2sbkvqkllc-findutils-4.6.0/bin
checking for references to /build in /nix/store/m7rga8yvy8kzw231qwggly2sbkvqkllc-findutils-4.6.0...
shrinking RPATHs of ELF executables and libraries in /nix/store/ycw5mdmk0kksyhj2kn7s3j3hplsdd289-findutils-4.6.0-info
strip is /nix/store/d1prcspbh2qsviipvnaxizcj8l3g7fpw-bootstrap-tools/bin/strip
checking for references to /build in /nix/store/ycw5mdmk0kksyhj2kn7s3j3hplsdd289-findutils-4.6.0-info...
building '/nix/store/ril503ayg0jgabwa2w6p572hrzn63790-stdenv-linux.drv'...

@orivej
Copy link
Contributor

orivej commented Jul 3, 2018

However, both ways will be visible in nix-shell, which may be as undesirable as hardening was before #28029.

I fail to get the meaning out of this sentence.

Sorry for being obscure! I imagine that a common scenario when CLICOLOR_FORCE is undesirable is when e.g. developers work in nix-shell, they expect that some-program | grep … | awk … produces plain output (without color escape codes), but CLICOLOR_FORCE makes some-program unexpectedly output colors to the pipe.


Any reason for sandbox not to allocate a pty (to enable colors for all builders in a generic way)?

This idea sounds good!

@orivej orivej requested a review from edolstra July 3, 2018 22:59
@infinisil
Copy link
Member Author

That's a good point, I haven't thought of that. @volth's idea of simulating (?) a terminal would certainly make it work correctly. An alternative not-as-nice solution to that would be to always unset CLICOLOR_FORCE in shellHook.

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: stdenv

Partial log (click to expand)

shrinking /nix/store/lqaq5639i7zza7rswlgp5k0kw8k03bns-findutils-4.6.0/bin/find
gzipping man pages under /nix/store/lqaq5639i7zza7rswlgp5k0kw8k03bns-findutils-4.6.0/share/man/
strip is /nix/store/gfgczbs0cy0blibb0acv39cayq7qbplg-bootstrap-tools/bin/strip
stripping (with command strip and flags -S) in /nix/store/lqaq5639i7zza7rswlgp5k0kw8k03bns-findutils-4.6.0/libexec  /nix/store/lqaq5639i7zza7rswlgp5k0kw8k03bns-findutils-4.6.0/bin
checking for references to /build in /nix/store/lqaq5639i7zza7rswlgp5k0kw8k03bns-findutils-4.6.0...
shrinking RPATHs of ELF executables and libraries in /nix/store/74m418038zpyh089rmk25p2rklp58hmp-findutils-4.6.0-info
strip is /nix/store/gfgczbs0cy0blibb0acv39cayq7qbplg-bootstrap-tools/bin/strip
checking for references to /build in /nix/store/74m418038zpyh089rmk25p2rklp58hmp-findutils-4.6.0-info...
building '/nix/store/yrn2ngglkaiy3dl5kc93gjvlbk1m5xix-stdenv-linux.drv'...
/nix/store/kfnsgi66dx0d6sab0xf3nv3r9zdd24jd-stdenv-linux

@edolstra
Copy link
Member

edolstra commented Jul 4, 2018

The main issue with this is that none of the current web interfaces for viewing Nix build logs know what to do with ANSI escape codes. It will probably show up as something like

�[1;31mfnord�[0m

when browsing a log in S3.

Nixpkgs used to use ANSI escapes heavily (for nested build logs) but people always complained about this so I removed it.

@infinisil
Copy link
Member Author

@edolstra It is however rather easy to strip color (https://stackoverflow.com/q/6534556/6605742), the other way is impossible, you can't add color once it's gone.

Also this would allow colors in online viewers (by translating the codes to html)

@edolstra
Copy link
Member

edolstra commented Jul 4, 2018

We can't strip or do any kind of processing when serving a log directly from S3.

@infinisil
Copy link
Member Author

@edolstra I don't exactly know how logs get there in the first place, is there no command it uses to produce the logs?

Oh and this just occured to me, an alternative which sounds much better: the Nix commands themselves could detect whether they output to a tty and strip color when it's not, along with full https://bixense.com/clicolors/ support. This would automatically propagate all colors when they would be there running the commands like normal.

@infinisil
Copy link
Member Author

I'll close this for now, I think this would need some support from Nix itself to properly and nicely work (see comment above), for which I currently neither have the time or knowledge to implement it.

@infinisil infinisil closed this Aug 4, 2018
@infinisil infinisil deleted the cmake-colors branch August 4, 2018 21:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants