Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pythonPackages.pytest-timeout: disable flaky test #80718

Merged
merged 1 commit into from Feb 22, 2020
Merged

pythonPackages.pytest-timeout: disable flaky test #80718

merged 1 commit into from Feb 22, 2020

Conversation

rnhmjoj
Copy link
Contributor

@rnhmjoj rnhmjoj commented Feb 21, 2020

Motivation for this change

ZHF: #80379

I could finally reproduce the failure on hydra by running the test in a i686-linux VM with stress -c 5. Only this test seems to fail reliably, so let's disable it!

Things done
  • Tested using sandboxing (nix.useSandbox)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@ofborg ofborg bot requested review from costrouc and makefu February 21, 2020 12:18
KamilaBorowska
KamilaBorowska reviewed Feb 21, 2020
View reviewed changes
Copy link
Member

@KamilaBorowska KamilaBorowska left a comment •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think test_fix_setup_func_only, test_fix_finalizer_func_only, test_ini_timeout_func_only and test_timeout_marker_inheritance are also flaky (yes, that's a lot, but as I have said, "it's kinda the entire point of the module to make tests flaky").

However, I will admit that test_suppresses_timeout_when_pdb_is_entered is most likely to cause issues as it's doing the most processing (which can cause the scheduler to schedule away), increasing the probability of flakiness being discovered. IMO all of those tests should be disabled, but disabling test_suppresses_timeout_when_pdb_is_entered is a big improvement still.

KamilaBorowska
KamilaBorowska suggested changes Feb 21, 2020
View reviewed changes
Copy link
Member

@KamilaBorowska KamilaBorowska left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please change the commit description to say pytest-timeout, not pytes-timeout.

@KamilaBorowska
Copy link
Member

@GrahamcOfBorg build python27Packages.pytest-timeout python37Packages.pytest-timeout python38Packages.pytest-timeout

@rnhmjoj rnhmjoj changed the title pythonPackages.pytes-timeout: disable flaky test pythonPackages.pytest-timeout: disable flaky test Feb 21, 2020
@KamilaBorowska
Copy link
Member

@GrahamcOfBorg build python27Packages.pytest-timeout python37Packages.pytest-timeout python38Packages.pytest-timeout

@rnhmjoj
Copy link
Contributor Author

rnhmjoj commented Feb 21, 2020

I think test_fix_setup_func_only, test_fix_finalizer_func_only, test_ini_timeout_func_only and test_timeout_marker_inheritance are also flaky (yes, that's a lot, but as I have said, "it's kinda the entire point of the module to make tests flaky").

I would say to try disabling this one first and see how hydra responds.

@rnhmjoj
Copy link
Contributor Author

rnhmjoj commented Feb 21, 2020

Ouch: mass rebuild.

@rnhmjoj rnhmjoj changed the base branch from master to staging-20.03 February 21, 2020 21:16
@rnhmjoj rnhmjoj changed the base branch from staging-20.03 to staging February 21, 2020 21:17
@KamilaBorowska
Copy link
Member

KamilaBorowska commented Feb 21, 2020 •
edited

Mass rebuild is fine, this is mostly Python code (so almost no compilation), so it should take an hour or so.

@rnhmjoj
Copy link
Contributor Author

rnhmjoj commented Feb 21, 2020

Do you think it's ok to push to master directly?

@KamilaBorowska
Copy link
Member

Yes.

@rnhmjoj rnhmjoj changed the base branch from staging to master February 22, 2020 08:32
@rnhmjoj rnhmjoj merged commit 57bcca3 into NixOS:master Feb 22, 2020
@rnhmjoj
Copy link
Contributor Author

rnhmjoj commented Feb 22, 2020

Backported in 142dd6c. Let's see.

@rnhmjoj rnhmjoj deleted the timeout branch April 7, 2020 12:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@KamilaBorowska KamilaBorowska KamilaBorowska requested changes

@FRidh FRidh Awaiting requested review from FRidh

@jonringer jonringer Awaiting requested review from jonringer

@costrouc costrouc Awaiting requested review from costrouc

@makefu makefu Awaiting requested review from makefu

Assignees
No one assigned
Labels
6.topic: python 10.rebuild-darwin: 501-1000 10.rebuild-darwin: 501+ 10.rebuild-linux: 501-1000 10.rebuild-linux: 501+
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@rnhmjoj @KamilaBorowska