Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pythonPackages.pycurl: disable flakey memory tests #72039

Closed
wants to merge 1 commit into from
Closed

pythonPackages.pycurl: disable flakey memory tests #72039

wants to merge 1 commit into from

Conversation

jonringer
Copy link
Contributor

@jonringer jonringer commented Oct 26, 2019 •
edited

Motivation for this change

fixes #71975

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
Notify maintainers

cc @

@jonringer jonringer added the 9.needs: port to stable A PR needs a backport to the stable release. label Oct 26, 2019
@jonringer jonringer requested a review from FRidh as a code owner October 26, 2019 17:45
@jonringer
Copy link
Contributor Author

for the port to stable, you may need to cherry-pick a few commits:

$ git log pkgs/development/python-modules/pycurl/default.nix
commit a014874750ab35ceb7dad3adc42bceef3a551029 (HEAD -> fix-pycurl, jonringer/fix-pycurl)
Author: Jonathan Ringer <jonringer117@gmail.com>
Date:   Sat Oct 26 10:44:00 2019 -0700

    pythonPackages.pycurl: disable flakey memory tests

commit 641148260cdde296ea67fea4b6d979bd87d577a8
Author: Jonathan Ringer <jonringer117@gmail.com>
Date:   Thu Oct 17 14:44:05 2019 -0700

    pythonPackages.pycurl: fix tests

commit d5beb9fb5626cd5442ec7549a8960becc48ef192
Author: Mario Rodas <marsam@users.noreply.github.com>
Date:   Thu Aug 22 16:20:00 2019 -0500

    pythonPackages.pycurl: fix build on darwin

@jonringer
Copy link
Contributor Author 

[23 built (4 failed), 59 copied (64.4 MiB), 14.8 MiB DL]
error: build of '/nix/store/6bf90vq6bm5p0m5lnqw7k9v16jda2b5l-env.drv' failed
https://github.com/NixOS/nixpkgs/pull/72039
4 package failed to build:
python27Packages.thumbor python37Packages.patator python38Packages.nvchecker python38Packages.patator

21 package were build:
pyCA python27Packages.blivet python27Packages.koji python27Packages.osc python27Packages.pycurl python27Packages.pykickstart python27Packages.rpkg python27Packages.urlgrabber python37Packages.bugwarrior python37Packages.koji python37Packages.nvchecker python37Packages.osc python37Packages.pycurl python37Packages.urlgrabber python38Packages.bugwarrior python38Packages.koji python38Packages.osc python38Packages.pycurl python38Packages.urlgrabber system-config-printer udocker

the 4 broken, are broken on master:

broken builds logs 
[10:52:07] jon@jon-desktop /home/jon/projects/nixpkgs (master)
$ nix build -f . --keep-going python27Packages.thumbor python37Packages.patator python38Packages.nvchecker python38Packages.patator
builder for '/nix/store/58sliyi6fi8dp54d0r5kish0ryl5rzfa-python3.8-pysnmp-4.4.12.drv' failed with exit code 1; last 10 log lines:
  building
  Executing setuptoolsBuildPhase
  Traceback (most recent call last):
    File "nix_run_setup", line 8, in <module>
      exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
    File "setup.py", line 66, in <module>
      observed_version = [int(x) for x in setuptools.__version__.split('.')]
    File "setup.py", line 66, in <listcomp>
      observed_version = [int(x) for x in setuptools.__version__.split('.')]
  ValueError: invalid literal for int() with base 10: 'post20191022'
cannot build derivation '/nix/store/18vd9b1pp7hi7x90v2szpndswm73n688-python3.8-patator-0.7.drv': 1 dependencies couldn't be built
builder for '/nix/store/49sixi010i4f6zk953fk0zg0xyikxc9x-python3.7-pysnmp-4.4.12.drv' failed with exit code 1; last 10 log lines:
  building
  Executing setuptoolsBuildPhase
  Traceback (most recent call last):
    File "nix_run_setup", line 8, in <module>
      exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
    File "setup.py", line 66, in <module>
      observed_version = [int(x) for x in setuptools.__version__.split('.')]
    File "setup.py", line 66, in <listcomp>
      observed_version = [int(x) for x in setuptools.__version__.split('.')]
  ValueError: invalid literal for int() with base 10: 'post20191022'
cannot build derivation '/nix/store/bdd7zm58hrm4zxgf785a7fx4c9z4ai59-python3.7-patator-0.7.drv': 1 dependencies couldn't be built
builder for '/nix/store/ckkriycpas609aah51vv4q17vbza8pz5-python2.7-remotecv-2.2.2.drv' failed with exit code 1; last 10 log lines:
    File "/nix/store/jcs9qj1knjghd07k8xm4mjm0qs12nkgd-python2.7-Pillow-6.2.0/lib/python2.7/site-packages/PIL/Image.py", line 930, in convert
      self.load()
    File "/nix/store/jcs9qj1knjghd07k8xm4mjm0qs12nkgd-python2.7-Pillow-6.2.0/lib/python2.7/site-packages/PIL/ImageFile.py", line 249, in load
      "(%d bytes not processed)" % len(b)
  IOError: image file is truncated (2 bytes not processed)

  ----------------------------------------------------------------------
  Ran 23 tests in 3.295s

  FAILED (errors=3)
cannot build derivation '/nix/store/ndvri99ba6vjvbkpf7cdasv0pkahcmld-python2.7-thumbor-6.7.0.drv': 1 dependencies couldn't be built
builder for '/nix/store/9wpzh2h9kyvwqj450y4h03hcl8y5ki6q-python3.8-python-rapidjson-0.8.0.drv' failed with exit code 1; last 10 log lines:
  E         + [123,123123123123123123123123]

  tests/test_enum.py:22: AssertionError
  =============================== warnings summary ===============================
  /nix/store/sv7c226g55y39ryxnrwxxdg01s184y1p-python3.8-pytest-5.2.1/lib/python3.8/site-packages/_pytest/mark/structures.py:321
    /nix/store/sv7c226g55y39ryxnrwxxdg01s184y1p-python3.8-pytest-5.2.1/lib/python3.8/site-packages/_pytest/mark/structures.py:321: PytestUnknownMarkWarning: Unknown pytest.mark.unit - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/latest/mark.html
      warnings.warn(

  -- Docs: https://docs.pytest.org/en/latest/warnings.html
  ======= 1 failed, 679 passed, 16 skipped, 2 xfailed, 1 warnings in 2.44s =======
cannot build derivation '/nix/store/z1lp4i112cidl030xz0k91kq1l40zrkg-python3.8-structlog-19.1.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/yyr7khz8g643y3b889jmmw2rmzldq7ys-python3.8-nvchecker-1.4.4.drv': 1 dependencies couldn't be built
[0 built (4 failed)]
error: build of '/nix/store/18vd9b1pp7hi7x90v2szpndswm73n688-python3.8-patator-0.7.drv', '/nix/store/bdd7zm58hrm4zxgf785a7fx4c9z4ai59-python3.7-patator-0.7.drv', '/nix/store/ndvri99ba6vjvbkpf7cdasv0pkahcmld-python2.7-thumbor-6.7.0.drv', '/nix/store/yyr7khz8g643y3b889jmmw2rmzldq7ys-python3.8-nvchecker-1.4.4.drv' failed

joepie91
joepie91 suggested changes Oct 26, 2019
View reviewed changes
Copy link
Contributor

@joepie91 joepie91 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will probably be insufficient, as there are other kinds of failing tests; see eg. #55945 (comment). I don't know if anyone has figured out a common cause yet, but if not, we should probably just disable the tests entirely so as to not break builds?

@jonringer
Copy link
Contributor Author

Even if it's insufficient, people are still having difficulties on it in a non-deterministic manner

@jonringer
Copy link
Contributor Author 

https://github.com/NixOS/nixpkgs/pull/72039
3 package are marked as broken and were skipped:
fedpkg python27Packages.nixpart python27Packages.nixpart0

4 package failed to build:
python27Packages.thumbor python37Packages.patator python38Packages.nvchecker python38Packages.patator

21 package were build:
pyCA python27Packages.blivet python27Packages.koji python27Packages.osc python27Packages.pycurl python27Packages.pykickstart python27Packages.rpkg python27Packages.urlgrabber python37Packages.bugwarrior python37Packages.koji python37Packages.nvchecker python37Packages.osc python37Packages.pycurl python37Packages.urlgrabber python38Packages.bugwarrior python38Packages.koji python38Packages.osc python38Packages.pycurl python38Packages.urlgrabber system-config-printer udocker

@joepie91
Copy link
Contributor

Right, I get that. My point is that those non-deterministic issues aren't solved by the current changes because they occur in other tests as well.

@jonringer
Copy link
Contributor Author

other than submitting an issue upstream, which some may be nixos specific, I'm not sure what to do

@joepie91
Copy link
Contributor

Disabling the install check is the workaround I've applied locally to make the build succeed, and perhaps that'd be the right thing to do here, at least until the root cause gets fixed.

It's perhaps a bit rigorous, but given that the failures occur in various completely unrelated tests, I'm not sure there are any other options - it looks to me like the problem is in how the tests get run (unintentional concurrency?), not in any one specific test, and so any non-concurrency-safe test might fail.

@jonringer
Copy link
Contributor Author

Closing, no desire currently to tackle this

@jonringer jonringer closed this Jan 9, 2020
@jonringer jonringer deleted the fix-pycurl branch January 9, 2020 19:57
@infinisil infinisil mentioned this pull request Jan 19, 2020
Closed
1 task
@jonringer jonringer restored the fix-pycurl branch January 19, 2020 20:18
@jonringer jonringer reopened this Jan 19, 2020
@jonringer
Copy link
Contributor Author

@GrahamcOfBorg build python3Packages.pycurl

@jonringer jonringer closed this Jan 19, 2020
@jonringer
Copy link
Contributor Author

this was merged, not sure why github just shows it as me pushing to master

@jonringer jonringer deleted the fix-pycurl branch January 19, 2020 20:31
@jonringer jonringer mentioned this pull request Jan 19, 2020
Merged
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@joepie91 joepie91 joepie91 requested changes

@FRidh FRidh Awaiting requested review from FRidh

Assignees
No one assigned
Labels
6.topic: python 9.needs: port to stable A PR needs a backport to the stable release. 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

pycurl test multi_memory_mgmt_test.py fails on 19.09 upgrade
2 participants
@jonringer @joepie91