Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pytest-testmon: skip broken tests #103786

Closed

Conversation

raboof
Copy link
Member

@raboof raboof commented Nov 14, 2020

Motivation for this change

For some reason the v1.0.3 PyPI tarball ships with tests that expect pytest-testmon master instead of the v1.0.3 code (tarpas/pytest-testmon#158), causing these tests to fail against 1.0.3 itself.

nixpkgs-review on unrelated PR's to staging currently fail because the tests for this package fail.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

For some reason the v1.0.3 PyPI tarball ships with tests that expect
pytest-testmon master instead of the v1.0.3 code.

tarpas/pytest-testmon#158
@dmvianna
Copy link
Contributor

Wouldn't it be better to wait for this issue to be sorted upstream before we included this (broken?) version in the nixpkgs?

@raboof
Copy link
Member Author

raboof commented Nov 14, 2020 via email

@dmvianna
Copy link
Contributor

If they're using semantic versioning, there would be no change in the API. I would not feel comfortable installing a test suite for which its own tests are failing. How would you know the library is functioning as expected and bugs were not introduced between 1.0.2 and 1.0.3?

@raboof
Copy link
Member Author

raboof commented Nov 16, 2020

If they're using semantic versioning, there would be no change in the API

Looks like they're not...

I would not feel comfortable installing a test suite for which its own tests are failing.

I agree it's a pretty bad look

How would you know the library is functioning as expected and bugs were not introduced between 1.0.2 and 1.0.3?

I manually inspected the test failures, and they are all failing because 1.0.3 does not contain the changes from the post-1.0.3 tarpas/pytest-testmon@cfb7c2d commit, which seem backwards-compatible.

Wouldn't it be better to wait for this issue to be sorted upstream before we included this (broken?) version in the nixpkgs?

That works for me as well, #103958

Copy link
Contributor

@dmvianna dmvianna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@raboof Chose reverting to 1.0.2 which passes its own tests. Doesn't look like any new functionality was added in 1.0.3 and tests are failing in that version.

@raboof
Copy link
Member Author

raboof commented Nov 16, 2020

(in fact I don't really care which solution we choose, I just want staging to build without failures again ;) )

@raboof raboof requested a review from dmvianna November 16, 2020 10:17
Copy link
Contributor

@dmvianna dmvianna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then let's go with 1.0.2

@mweinelt mweinelt closed this Nov 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants