Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

prometheus: disable tests #44864

Closed
wants to merge 1 commit into from
Closed

prometheus: disable tests #44864

wants to merge 1 commit into from

Conversation

lukateras
Copy link
Member

Motivation for this change

Tests silently fail on Prometheus 2.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: prometheus

Partial log (click to expand)

shrinking /nix/store/bm1xkipjpsws1y3snc94b43k0ab2hray-prometheus-1.8.2-bin/bin/promtool
strip is /nix/store/ah0va6j4cnwj9nx4j6rwcfc8nh785jwm-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/bm1xkipjpsws1y3snc94b43k0ab2hray-prometheus-1.8.2-bin/bin
patching script interpreter paths in /nix/store/bm1xkipjpsws1y3snc94b43k0ab2hray-prometheus-1.8.2-bin
checking for references to /build in /nix/store/bm1xkipjpsws1y3snc94b43k0ab2hray-prometheus-1.8.2-bin...
shrinking RPATHs of ELF executables and libraries in /nix/store/30dzdsrg2zxaw5gsrcx207qqg6gin28j-prometheus-1.8.2
strip is /nix/store/ah0va6j4cnwj9nx4j6rwcfc8nh785jwm-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/30dzdsrg2zxaw5gsrcx207qqg6gin28j-prometheus-1.8.2
checking for references to /build in /nix/store/30dzdsrg2zxaw5gsrcx207qqg6gin28j-prometheus-1.8.2...
/nix/store/bm1xkipjpsws1y3snc94b43k0ab2hray-prometheus-1.8.2-bin

@GrahamcOfBorg
Copy link

Success on x86_64-darwin (full log)

Attempted: prometheus

Partial log (click to expand)

installing
post-installation fixup
Moving /nix/store/gqgpk60kpghxzndx9v87av5cqz3q5c59-prometheus-1.8.2-bin/share/doc to /nix/store/smnljgdp7lkjwyy8qkqicj73xwaj3pm3-prometheus-1.8.2/share/doc
Removing empty /nix/store/gqgpk60kpghxzndx9v87av5cqz3q5c59-prometheus-1.8.2-bin/share/ and (possibly) its parents
strip is /nix/store/251fvx4mx9q4zarbca5cciasmn66p668-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/gqgpk60kpghxzndx9v87av5cqz3q5c59-prometheus-1.8.2-bin/bin
patching script interpreter paths in /nix/store/gqgpk60kpghxzndx9v87av5cqz3q5c59-prometheus-1.8.2-bin
strip is /nix/store/251fvx4mx9q4zarbca5cciasmn66p668-cctools-binutils-darwin/bin/strip
patching script interpreter paths in /nix/store/smnljgdp7lkjwyy8qkqicj73xwaj3pm3-prometheus-1.8.2
/nix/store/gqgpk60kpghxzndx9v87av5cqz3q5c59-prometheus-1.8.2-bin

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: prometheus

Partial log (click to expand)

shrinking /nix/store/4mwdycn4lliqsdc12xx4yl5qyk9jl64f-prometheus-1.8.2-bin/bin/promtool
shrinking /nix/store/4mwdycn4lliqsdc12xx4yl5qyk9jl64f-prometheus-1.8.2-bin/bin/storagetool
strip is /nix/store/gpc2wld1s0c6qzx9326cwn1wcx29xzsj-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/4mwdycn4lliqsdc12xx4yl5qyk9jl64f-prometheus-1.8.2-bin/bin
patching script interpreter paths in /nix/store/4mwdycn4lliqsdc12xx4yl5qyk9jl64f-prometheus-1.8.2-bin
checking for references to /build in /nix/store/4mwdycn4lliqsdc12xx4yl5qyk9jl64f-prometheus-1.8.2-bin...
shrinking RPATHs of ELF executables and libraries in /nix/store/c74xs1xacixdfwmf0j6nj1gkby4ajxny-prometheus-1.8.2
strip is /nix/store/gpc2wld1s0c6qzx9326cwn1wcx29xzsj-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/c74xs1xacixdfwmf0j6nj1gkby4ajxny-prometheus-1.8.2
checking for references to /build in /nix/store/c74xs1xacixdfwmf0j6nj1gkby4ajxny-prometheus-1.8.2...

@coretemp
Copy link
Contributor

I don't quite understand the rationale here. If tests fail, the solution should be to fix the tests, not disable them. If you don't want to fix the tests yourself, contact upstream.

Disabling the tests only seems to remove value of Nixpkgs.

@lukateras
Copy link
Member Author

That's not helpful. Tests in Nixpkgs are disabled due to failure, breakage, etc. all the time, search for "doCheck = false;". At the moment, package is broken and people running Prometheus 2 can't rebuild systems anymore.

@coretemp
Copy link
Contributor

I know about the practice and I don't agree with it.

@samueldr
Copy link
Member

I'm not sure I understand the issue here; I may be overlooking something or misunderstanding.

prometheus_2 builds (seemingly) finelatest build on hydra as of now, but does have an ignored failure in the log:

=== RUN   TestRecoverParserRuntime
parser panic: runtime error: index out of range
[...]
vector result 0 metric
--- PASS: TestEvaluations (4.81s)
PASS

You say:

At the moment, package is broken and people running Prometheus 2 can't rebuild systems anymore.

How are they unable to rebuild their systems if the package (prometheus_2) itself builds fine even if there is a failing test?

I have verified, also, that the package builds locally by forcing a rebuild, and it does build without failure (with the silently failing test).

IF the issue is that prometheus fails to run because of the issue causing the silently failing test, the better approach would be to ensure the test fails the build and not to disable all tests, right?

Additionally, when there is a failure in one test from upstream, I think the more common approach is to disable that test only through a project or framework specific method (e.g. a patch, sed the test out or rm the files).

@lukateras
Copy link
Member Author

Thanks, @samueldr! It looks like this has been resolved in the meantime.

@lukateras lukateras closed this Oct 12, 2018
@lukateras lukateras deleted the yegortimoshenko-patch-1 branch October 12, 2018 19:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@lukateras @GrahamcOfBorg @coretemp @samueldr