buildbot: Python 3 support and other improvements #45890

lopsided98 · 2018-09-01T01:47:56Z

Motivation for this change

The latest versions of pylint and astroid do not support Python 2, which breaks buildbot (#45782) because it only supports Python 2 in nixpkgs. Buildbot supports Python 3, but this is potentially a breaking change because buildbot uses Python in its configuration files. Therefore, this PR adds configurable support for Python 2 or 3, which Python 2 being the default. The 1.0 branches of pylint and astroid are still supported until Python 2's EOL, so they are used to build buildbot with Python 2.

This PR supersedes #45789.

Things done

First, I updated and added Python 3 support to sphinx-jinja. This mainly required fixing how the tests were run. This commit supersedes #45763.

I updated tempita to use a fork that adds Python 3 support. The upstream tempita package appears to be abandoned. This fork is used by the official Arch Linux package of tempita, but is somewhat undesirable because it is not an official release. This commit supersedes #39692.

I updated astroid and pylint to their latest versions, and added astroid1 and pylint1 packages which use the still maintained 1.0 branch of these packages. Because there are a number of different ways to implement packages with multiple versions in Nix, I'm not sure if I chose the best way.

While I was testing I ran into #40273, which is supposedly fixed but does not seem to be (see #40273 (comment)). To work around this I changed hyperlink to not use pytest to run its tests (the default checkPhase seems to work fine) and made some changes to autobahn (which do not fix the problem, but only one package needs to be fixed for the conflict to be resolved). These changes are not critical, because they should not be necessary once #40273 is truly fixed, but I included them anyway because I believe they result in cleaner derivations.

I then updated all the components of buildbot to 1.3.0. buildbot-worker had already been updated by an automated update, but the rest of the components had not. This has happened a number of times recently and sometimes causes problems, especially when buildbot-pkg is updated without updating the plugins.

The last commit adds Python 3 support as well as a number of other improvements that I have been running locally for about a year. There are no intentional breaking changes, but I have not tested every possible scenario.

All the buildbot packages are moved to pythonPackages. A new python attribute is added to each package's passthru, allowing the NixOS module to determine which version of Python to use. The tests are currently disabled for both Python 2 and 3, because a test fails due to what appears to be a bug in buildbot. I attempted to disable the failing test, but this simply causes the error to move to a different test and so on. The error message (see comment above doCheck = false) relates to a recently added buildbot feature, so this points to a problem with buildbot.

I changed how the plugin wrapper works, making the wrapped package into a Python module that can be added to a Python environment using python.withPackages.

In the buildbot-master NixOS module, I switched to directly invoking twistd (as was already done with buildbot-worker) in order to log to stdout and therefore the systemd journal, rather than a file. I also added a new pythonPackages option, which takes a function that is passed to python.withPackages and is used to set the service's PYTHONPATH. I also generate the twisted tac file within the module, rather than using create-master (create-master is still necessary to create the database). This gives more control over the initialization process (especially useful for buildbot-worker).

In the buildbot-worker NixOS module, I made similar improvements to the PYTHONPATH handling, but did not add a pythonPackages option because workers never run user Python code AFAIK. I also generate the tac file in the module, which makes it possible to implement the workerPassFile option (checks one of the boxes in #24288). I also added hostMessage and adminMessage options, which allow users to declaratively set those messages.

The buildbot NixOS test passes, including the commented out daemon test, which supposedly was previously broken. I configured the tests to run with both Python 2 and 3.

I would appreciate it if other buildbot users would test this PR. I have only run the tests and used it with my buildbot deployment, which has a number of customizations with might obscure bugs.

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

xeji · 2018-09-01T10:47:39Z

cc @dotlambda

nixos/tests/buildbot.nix

pkgs/top-level/python-packages.nix

dotlambda · 2018-09-01T10:59:23Z

pkgs/development/tools/build-managers/buildbot/default.nix

-    ln -sfv ${package}/lib $out/lib
-  '';
+  # Can't use runCommand because it does not support propagatedBuildInputs
+  withPlugins = plugins: toPythonModule (stdenv.mkDerivation {


Why don't you use buildPythonPackage?

This can simply be python.withPackages.

pkgs/development/tools/build-managers/buildbot/default.nix

pkgs/development/python-modules/sphinx-jinja/default.nix

pkgs/development/python-modules/autobahn/default.nix

FRidh

I'm not sure whether we really should move buildbot into python-packages, however, if we do this, then the expressions should live there as well (python-modules/), and more importantly, no expressions should use buildPythonApplication.

pkgs/development/python-modules/astroid/1.6.nix

FRidh · 2018-09-01T11:12:12Z

pkgs/development/python-modules/autobahn/default.nix

  propagatedBuildInputs = [ six twisted zope_interface txaio ] ++
    (stdenv.lib.optional isPy33 asyncio) ++
    (stdenv.lib.optionals (!isPy3k) [ trollius futures ]);

+  # Upstream claim python2 support, but tests require pytest-asyncio which


trollius 2 is needed for asyncio support

pkgs/development/python-modules/pylint/1.9.nix

FRidh · 2018-09-01T11:14:11Z

pkgs/development/tools/build-managers/buildbot/default.nix

-    ln -sfv ${package}/lib $out/lib
-  '';
+  # Can't use runCommand because it does not support propagatedBuildInputs
+  withPlugins = plugins: toPythonModule (stdenv.mkDerivation {


This can simply be python.withPackages.

dotlambda · 2018-09-01T11:24:19Z

I'm not sure whether we really should move buildbot into python-packages

I still think that it is easier if we put it in pythonPackages because otherwise, multiple packages, e.g. buildbot, buildbot-worker, ... have to be overridden to use a different Python version.
However, I'm not sure whether we should put an alias in all-packages.nix.
Anyway, this needs a changelog entry.

lopsided98 · 2018-09-04T20:00:14Z

I think I have addressed all the issues that were brought up. I had to add a few new commits to fix autobahn's tests with Python 2, and a commit to add a patch to sqlalchemy_migrate to make it compatible with Python 3.6 (I previously had this in an overlay and forgot about it).

@nand0p I kept you as the maintainer of the 1.x branches of astroid and pylint. Is this okay?

Buildbot just updated to 1.4.0, and its tests now pass for Python 2 and 3.

lopsided98 · 2018-09-24T00:32:46Z

I rebased against master and moved the changelog entry to the 19.03 changelog.

nixos/modules/services/continuous-integration/buildbot/master.nix

nixos/modules/services/continuous-integration/buildbot/worker.nix

pkgs/development/python-modules/tempita/default.nix

pkgs/top-level/all-packages.nix

infinisil · 2018-10-11T21:58:03Z

@GrahamcOfBorg test buildbot.python2 buildbot.python3

GrahamcOfBorg · 2018-10-11T22:14:29Z

Success on aarch64-linux (full log)

Attempted: tests.buildbot.python2, tests.buildbot.python3

Partial log (click to expand)

Oct 11 22:13:51 bbworker mqa4mvwqafrn53kcwzvxysr4p5llcpmx-unit-script-buildbot-worker-pre-start[753]: mkdir: created directory '/home/bbworker/worker/info'
Oct 11 22:13:51 bbworker systemd[1]: Started Buildbot Worker..
total 12
drwxr-xr-x 3 bbworker bbworker 4096 Oct 11 22:13 .
drwx------ 3 bbworker bbworker 4096 Oct 11 22:13 ..
drwxr-xr-x 2 bbworker bbworker 4096 Oct 11 22:13 info
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
/nix/store/7mnda9bfn2czxnhw4p42hbw4gviwdhb8-vm-test-run-buildbot
/nix/store/qja43p34slaqchzr0pdr5jcd8s90szjn-vm-test-run-buildbot

GrahamcOfBorg · 2018-10-11T22:59:10Z

Success on x86_64-linux (full log)

Attempted: tests.buildbot.python2, tests.buildbot.python3

Partial log (click to expand)

Oct 11 22:58:14 bbworker systemd[1]: Starting Buildbot Worker....
Oct 11 22:58:14 bbworker systemd[1]: Started Buildbot Worker..
total 12
drwxr-xr-x 3 bbworker bbworker 4096 Oct 11 22:58 .
drwx------ 3 bbworker bbworker 4096 Oct 11 22:58 ..
drwxr-xr-x 2 bbworker bbworker 4096 Oct 11 22:58 info
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
/nix/store/2p09g21258a4dahpyqr9a598gpaif8ij-vm-test-run-buildbot
/nix/store/skbj8hg7in1581fhzm0wll0ycc6w4dd1-vm-test-run-buildbot

lopsided98 · 2018-10-12T02:14:26Z

@infinisil I fixed the issues you brought up.

Should this go into staging?

Also, there may be some buildbot users on 18.03 who will find their configs broken when they upgrade to 18.09, so we may want to backport this eventually. I'm not sure if we would backport the whole PR, or just the minimum to fix the build.

infinisil · 2018-10-12T12:06:27Z

I think master is alright. Feel free to open another PR with a backport if you feel it's necessary.

Thanks for the PR, the NixOS test is very nice too :)

lopsided98 requested a review from FRidh as a code owner September 1, 2018 01:47

lopsided98 mentioned this pull request Sep 1, 2018

buildbot: use python3Packages #45789

Closed

9 tasks

GrahamcOfBorg added 6.topic: nixos 6.topic: python 8.has: module (update) 8.has: clean-up 8.has: package (new) 10.rebuild-darwin: 11-100 10.rebuild-linux: 101-500 labels Sep 1, 2018

dotlambda reviewed Sep 1, 2018

View reviewed changes

pkgs/development/python-modules/autobahn/default.nix Outdated Show resolved Hide resolved

FRidh requested changes Sep 1, 2018

View reviewed changes

lopsided98 force-pushed the buildbot-python3 branch 2 times, most recently from 856b4b4 to 6586375 Compare September 3, 2018 04:13

GrahamcOfBorg added 8.has: changelog 8.has: documentation 2.status: merge conflict labels Sep 3, 2018

lopsided98 force-pushed the buildbot-python3 branch 3 times, most recently from 73c3ff0 to 87a3737 Compare September 4, 2018 19:15

GrahamcOfBorg removed the 2.status: merge conflict label Sep 4, 2018

lopsided98 force-pushed the buildbot-python3 branch from 87a3737 to 2977cd6 Compare September 4, 2018 19:20

GrahamcOfBorg added 10.rebuild-darwin: 101-500 and removed 10.rebuild-darwin: 11-100 labels Sep 4, 2018

lopsided98 force-pushed the buildbot-python3 branch 2 times, most recently from 69fb188 to c99bfcb Compare September 6, 2018 05:13

GrahamcOfBorg added the 10.rebuild-darwin: 11-100 label Sep 6, 2018

lopsided98 force-pushed the buildbot-python3 branch from ff936f2 to b56e346 Compare September 24, 2018 00:28

lopsided98 force-pushed the buildbot-python3 branch 5 times, most recently from cf05986 to 86c10a7 Compare October 11, 2018 20:55

infinisil requested changes Oct 11, 2018

View reviewed changes

pythonPackages.sphinx-jinja: 0.2.1 -> 1.1.0, Python 3 support

225b8de

lopsided98 added 8 commits October 11, 2018 18:14

pythonPackages.tempita: 0.5.2 -> 0.5.3-2016-09-28

511d651

python2Packages.astroid: init at 1.6.5

4f09e05

python2Packages.pylint: init at 1.9.2

67ea567

pythonPackages.hyperlink: don't create .pytest_cache

fcaae4f

pythonPackages.trollius: 1.0.4 -> 2.2

820fecc

pythonPackages.autobahn: use checkInputs

1c3f730

pythonPackages.sqlalchemy_migrate: patch backslash escapes

5b2a1db

buildbot: 1.2.0 -> 1.4.0

71c4246

buildbot: add Python 3 support

73c523a

lopsided98 force-pushed the buildbot-python3 branch from 86c10a7 to 73c523a Compare October 12, 2018 02:04

infinisil approved these changes Oct 12, 2018

View reviewed changes

infinisil merged commit c26d600 into NixOS:master Oct 12, 2018

This was referenced Oct 12, 2018

sphinx-jinja: 0.2.1 -> 1.1.0, Python 3 support #45763

Closed

tempita: 0.5.2 -> 0.5.3-2016-09-28, Python 3 support #39692

Closed

lopsided98 deleted the buildbot-python3 branch October 12, 2018 13:19

lopsided98 mentioned this pull request Dec 13, 2018

buildbot broken by pylint upgrade #45782

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

buildbot: Python 3 support and other improvements #45890

buildbot: Python 3 support and other improvements #45890

lopsided98 commented Sep 1, 2018 •

edited

xeji commented Sep 1, 2018

dotlambda Sep 1, 2018

FRidh Sep 1, 2018

FRidh left a comment •

edited

FRidh Sep 1, 2018

FRidh Sep 1, 2018

dotlambda commented Sep 1, 2018

lopsided98 commented Sep 4, 2018

lopsided98 commented Sep 24, 2018

infinisil commented Oct 11, 2018

GrahamcOfBorg commented Oct 11, 2018

GrahamcOfBorg commented Oct 11, 2018

lopsided98 commented Oct 12, 2018

infinisil commented Oct 12, 2018

buildbot: Python 3 support and other improvements #45890

buildbot: Python 3 support and other improvements #45890

Conversation

lopsided98 commented Sep 1, 2018 • edited

Motivation for this change

Things done

xeji commented Sep 1, 2018

dotlambda Sep 1, 2018

Choose a reason for hiding this comment

FRidh Sep 1, 2018

Choose a reason for hiding this comment

FRidh left a comment • edited

Choose a reason for hiding this comment

FRidh Sep 1, 2018

Choose a reason for hiding this comment

FRidh Sep 1, 2018

Choose a reason for hiding this comment

dotlambda commented Sep 1, 2018

lopsided98 commented Sep 4, 2018

lopsided98 commented Sep 24, 2018

infinisil commented Oct 11, 2018

GrahamcOfBorg commented Oct 11, 2018

GrahamcOfBorg commented Oct 11, 2018

lopsided98 commented Oct 12, 2018

infinisil commented Oct 12, 2018

lopsided98 commented Sep 1, 2018 •

edited

FRidh left a comment •

edited