python3Packages.sopel: 7.0.5 -> 7.0.6 #95689

mweinelt · 2020-08-17T13:00:00Z

Motivation for this change

Noticed it was broken in #95673 (comment).

Things done

mweinelt · 2020-08-17T13:00:42Z

I'm currently having issues with the test suite and am looking for help. Reported the issue upstream at sopel-irc/sopel#1926.

============================= test session starts ==============================
platform linux -- Python 3.8.5, pytest-5.4.3, py-1.8.2, pluggy-0.13.1 -- /nix/store/fjgnz0xfl04hsblsi4ym5y5akfh6mlmy-python3-3.8.5/bin/python3.8
cachedir: .pytest_cache
rootdir: /build/sopel-7.0.6
plugins: sopel-7.0.6
collected 215 items / 1 error / 214 selected

==================================== ERRORS ====================================
_____________________ ERROR collecting sopel/test_tools.py _____________________
import file mismatch:
imported module 'sopel.test_tools' has this __file__ attribute:
  /nix/store/apl4l88a2vy1d7m6d46ncimcqgcz9xqa-python3.8-sopel-7.0.6/lib/python3.8/site-packages/sopel/test_tools.py
which is not the same as the test file we want to collect:
  /build/sopel-7.0.6/sopel/test_tools.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
=========================== short test summary info ============================
ERROR sopel/test_tools.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 1.07s ===============================

drewrisinger · 2020-08-18T02:59:08Z

@mweinelt Here's what I think is happening:

Nix builds sopel package
Nix installs sopel package to $out
checkPhase runs pytest in $sourceRoot
Pytest finds "tests" in ./sopel && ./test. "Test" in ./sopel is a mistake due to naming convention, test_tools.py registers as a file that pytest wants to run (matches test_*.py pattern). This part could be fixed by running pytest ./test in checkPhase.
file conflict is due to finding sopel/test_tools.py in two file locations: $out/.../sopel, and $sourceRoot (actually $TMP/$sourceRoot)/sopel/...

An aside, I generally recommend using pytestCheckHook:

{
...
, pytestCheckHook
, ...
}:

buildPythonPackage {
  checkInputs = [ pytestCheckHook];
  pytestFlagsArray = [ "./test" ]; # prevent finding "test" in ./sopel/test_tools.py
}

If that doesn't work, my other suggestion is make a temp dir and move the tests to that:

(using pytestCheckHook as above)
preCheck = ''
  export TESTDIR=$(mktemp -d)
  cp -r ./test $TESTDIR
  pushd $TESTDIR
'';
postCheck = "popd";

mweinelt · 2020-08-18T10:16:49Z

Thanks, that explanation helped!

mweinelt · 2020-08-18T12:11:16Z

No space left on device 🤷

drewrisinger · 2020-08-18T14:40:23Z

No space left on device 🤷

Have you run nix-collect-garbage recently? I keep freeing up 10s of GBs using that from running nix-review all the time.

A few packages don't build with this as-is due to mocket build failures:

$ nix-review pr 95689:

https://github.com/NixOS/nixpkgs/pull/95689
4 packages failed to build:
python27Packages.mocket python37Packages.geoip2 python37Packages.mocket python37Packages.sopel

6 packages built:
python37Packages.maxminddb python38Packages.geoip2 python38Packages.maxminddb python38Packages.mocket python38Packages.sopel zeronet

Mocket might be able to be disabled on python2:

{ ...
, isPy3k
, ...
}:

buildPythonPackage {
  disabled = ! isPy3k;
}

Last log lines from failed mocket builds:

builder for '/nix/store/k34nyxsvlihrx17sd7scz84sz3df8r61-python2.7-mocket-3.8.7.drv' failed with exit code 1; last 10 log lines:
      replace_conflicting=replace_conflicting
    File "/nix/store/gz139sqlwiyycgbw1hws53q1c2v2vfa4-python2.7-setuptools-44.0.0/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1065, in best_match
      return self.obtain(req, installer)
    File "/nix/store/gz139sqlwiyycgbw1hws53q1c2v2vfa4-python2.7-setuptools-44.0.0/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1077, in obtain
      return installer(requirement)
    File "/nix/store/gz139sqlwiyycgbw1hws53q1c2v2vfa4-python2.7-setuptools-44.0.0/lib/python2.7/site-packages/setuptools/dist.py", line 777, in fetch_build_egg
      return fetch_build_egg(self, req)
    File "/nix/store/gz139sqlwiyycgbw1hws53q1c2v2vfa4-python2.7-setuptools-44.0.0/lib/python2.7/site-packages/setuptools/installer.py", line 130, in fetch_build_egg
      raise DistutilsError(str(e))
  distutils.errors.DistutilsError: Command '['/nix/store/6xyf4rb6ch0xgf6c7x2p0g6l6k76yqn2-python-2.7.18/bin/python2.7', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/build/tmpzo20Gd', '--quiet', 'pipenv']' returned non-zero exit status 1
builder for '/nix/store/pqh0as41fqx8qqa2gih7rnz1wl4lawdf-python3.7-mocket-3.8.7.drv' failed with exit code 1; last 10 log lines:
      replace_conflicting=replace_conflicting
    File "/nix/store/z00af9n2g82fiw3yqhdzhh6bm90x5b3n-python3.7-setuptools-47.3.1/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1065, in best_match
      return self.obtain(req, installer)
    File "/nix/store/z00af9n2g82fiw3yqhdzhh6bm90x5b3n-python3.7-setuptools-47.3.1/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1077, in obtain
      return installer(requirement)
    File "/nix/store/z00af9n2g82fiw3yqhdzhh6bm90x5b3n-python3.7-setuptools-47.3.1/lib/python3.7/site-packages/setuptools/dist.py", line 754, in fetch_build_egg
      return fetch_build_egg(self, req)
    File "/nix/store/z00af9n2g82fiw3yqhdzhh6bm90x5b3n-python3.7-setuptools-47.3.1/lib/python3.7/site-packages/setuptools/installer.py", line 130, in fetch_build_egg
      raise DistutilsError(str(e))
  distutils.errors.DistutilsError: Command '['/nix/store/9hdzs3g6rb76grwzlsbxyxwxg2n0fl4a-python3-3.7.8/bin/python3.7', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/build/tmpxdi3ysc3', '--quiet', 'pipenv']' returned non-zero exit status 1.

mweinelt · 2020-08-18T15:02:59Z

Have you run nix-collect-garbage recently? I keep freeing up 10s of GBs using that from running nix-review all the time.

I'm talking about ofBorg :)

Looking into the other issues asap.

mweinelt · 2020-08-18T15:33:43Z

Sorted.

Result of nixpkgs-review pr 95689 1

10 packages built:

- python27Packages.mocket
- python37Packages.geoip2
- python37Packages.maxminddb
- python37Packages.mocket
- python37Packages.sopel
- python38Packages.geoip2
- python38Packages.maxminddb
- python38Packages.mocket
- python38Packages.sopel
- zeronet

jonringer

diff LGTM

https://github.com/NixOS/nixpkgs/pull/95689
10 packages built:
python27Packages.mocket python37Packages.geoip2 python37Packages.maxminddb python37Packages.mocket python37Packages.sopel python38Packages.geoip2 python38Packages.maxminddb python38Packages.mocket python38Packages.sopel zeronet

jonringer · 2020-08-18T18:01:41Z

is ofborg dead?

drewrisinger

See comments.

pkgs/development/python-modules/geoip2/default.nix

pkgs/development/python-modules/maxminddb/default.nix

pkgs/development/python-modules/mocket/default.nix

drewrisinger · 2020-08-18T18:12:19Z

pkgs/development/python-modules/mocket/default.nix

+  # Pypi has no runtests.py, github has no requirements.txt. No way to test, no way to install.
+  doCheck = false;


Maybe file an upstream issue?

Also, it looks like the runtest.py basically just runs pytest, and the tests are included in the PyPi sdist. So maybe just try running pytestCheckHook?

generally a tox.ini or the test_requires section of setup.py will mention what is needed.

However, this does a non-standard construction. Needs xxhash aiohttp and async-timeout
https://github.com/mindflayer/python-mocket/blob/fb24872ae2a530dad0e645e3e9c39dcea29e999d/runtests.py#L22

Testing mocket requires packaging of pook, which requires mocket for testing.

Boom. Infinite loop.

I only ever wanted to fixup those packages, not maintain them 😞

drewrisinger · 2020-08-18T18:13:08Z

pkgs/development/python-modules/sopel/default.nix

+  postCheck = ''
+    popd
  '';


Suggested change

postCheck = ''

popd

'';

postCheck = "popd";

pkgs/development/python-modules/geoip2/default.nix

pkgs/development/python-modules/sopel/default.nix

mweinelt

is ofborg dead?

@jonringer Yeah, graham is on vacation and cole and LnL apparently can't fix this.

jonringer · 2020-08-18T19:17:07Z

@jonringer Yeah, graham is on vacation and cole and LnL apparently can't fix this.

build and test command seems to work. just eval isn't working... which I guess is what it normally does xD

mweinelt · 2020-08-18T20:27:41Z

Result of nixpkgs-review pr 95689 1

10 packages built:

- python27Packages.mocket
- python37Packages.geoip2
- python37Packages.maxminddb
- python37Packages.mocket
- python37Packages.sopel
- python38Packages.geoip2
- python38Packages.maxminddb
- python38Packages.mocket
- python38Packages.sopel
- zeronet

jonringer

LGTM

Result of nixpkgs-review pr 95689 1

10 packages built:

- python27Packages.mocket
- python37Packages.geoip2
- python37Packages.maxminddb
- python37Packages.mocket
- python37Packages.sopel
- python38Packages.geoip2
- python38Packages.maxminddb
- python38Packages.mocket
- python38Packages.sopel
- zeronet

pipenv should not be required for sdist installations. noticed this behavior when reviewing NixOS/nixpkgs#95689

mweinelt changed the title ~~sopel: 7.0.5 -> 7.0.6~~ python3Packages.sopel: 7.0.5 -> 7.0.6 Aug 17, 2020

ofborg bot added 6.topic: python 8.has: clean-up 8.has: package (new) labels Aug 17, 2020

ofborg bot requested a review from mogorman August 17, 2020 14:36

ofborg bot added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Aug 17, 2020

mweinelt force-pushed the sopel branch from e445113 to b7994d1 Compare August 18, 2020 10:11

mweinelt marked this pull request as ready for review August 18, 2020 10:16

mweinelt requested review from FRidh and jonringer as code owners August 18, 2020 10:16

mweinelt mentioned this pull request Aug 18, 2020

Import file mismatch in testsuite sopel-irc/sopel#1926

Closed

python3Packages.maxminddb: 1.5.4 -> 2.0.2

939b6f3

mweinelt force-pushed the sopel branch from b7994d1 to cd6facb Compare August 18, 2020 15:23

jonringer approved these changes Aug 18, 2020

View reviewed changes

drewrisinger suggested changes Aug 18, 2020

View reviewed changes

mweinelt commented Aug 18, 2020

View reviewed changes

pythonPackages.mocket: init at 3.8.7

8db57d9

mweinelt force-pushed the sopel branch 3 times, most recently from c0ea40b to a36b856 Compare August 18, 2020 20:23

mweinelt added 2 commits August 18, 2020 22:25

python3Packages.geoip2: 3.0.0 -> 4.0.2

9c767bc

python3Packages.sopel: 7.0.5 -> 7.0.6

e83abaf

mweinelt force-pushed the sopel branch from a36b856 to e83abaf Compare August 18, 2020 20:25

jonringer approved these changes Aug 18, 2020

View reviewed changes

jonringer merged commit 2addc08 into NixOS:master Aug 18, 2020

mweinelt deleted the sopel branch August 18, 2020 21:18

jonringer added a commit to jonringer/python-mocket that referenced this pull request Aug 18, 2020

Remove pipenv from setup_requires

b283d68

pipenv should not be required for sdist installations. noticed this behavior when reviewing NixOS/nixpkgs#95689

mindflayer pushed a commit to mindflayer/python-mocket that referenced this pull request Aug 19, 2020

Remove pipenv from setup_requires (#121)

1ac521c

pipenv should not be required for sdist installations. noticed this behavior when reviewing NixOS/nixpkgs#95689

drewrisinger mentioned this pull request Aug 24, 2020

pylode: init at 2.8.6 #95863

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python3Packages.sopel: 7.0.5 -> 7.0.6 #95689

python3Packages.sopel: 7.0.5 -> 7.0.6 #95689

mweinelt commented Aug 17, 2020

mweinelt commented Aug 17, 2020

drewrisinger commented Aug 18, 2020

mweinelt commented Aug 18, 2020

mweinelt commented Aug 18, 2020

drewrisinger commented Aug 18, 2020

mweinelt commented Aug 18, 2020 •

edited

mweinelt commented Aug 18, 2020

jonringer left a comment

jonringer commented Aug 18, 2020

drewrisinger left a comment

drewrisinger Aug 18, 2020

jonringer Aug 18, 2020

mweinelt Aug 18, 2020

drewrisinger Aug 18, 2020

mweinelt left a comment •

edited

jonringer commented Aug 18, 2020

mweinelt commented Aug 18, 2020

jonringer left a comment

		# Pypi has no runtests.py, github has no requirements.txt. No way to test, no way to install.
		doCheck = false;

python3Packages.sopel: 7.0.5 -> 7.0.6 #95689

python3Packages.sopel: 7.0.5 -> 7.0.6 #95689

Conversation

mweinelt commented Aug 17, 2020

Motivation for this change

Things done

mweinelt commented Aug 17, 2020

drewrisinger commented Aug 18, 2020

mweinelt commented Aug 18, 2020

mweinelt commented Aug 18, 2020

drewrisinger commented Aug 18, 2020

mweinelt commented Aug 18, 2020 • edited

mweinelt commented Aug 18, 2020

jonringer left a comment

Choose a reason for hiding this comment

jonringer commented Aug 18, 2020

drewrisinger left a comment

Choose a reason for hiding this comment

drewrisinger Aug 18, 2020

Choose a reason for hiding this comment

jonringer Aug 18, 2020

Choose a reason for hiding this comment

mweinelt Aug 18, 2020

Choose a reason for hiding this comment

drewrisinger Aug 18, 2020

Choose a reason for hiding this comment

mweinelt left a comment • edited

Choose a reason for hiding this comment

jonringer commented Aug 18, 2020

mweinelt commented Aug 18, 2020

jonringer left a comment

Choose a reason for hiding this comment

mweinelt commented Aug 18, 2020 •

edited

mweinelt left a comment •

edited