Add setup-hooks for scons and waf #50293
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@GrahamcOfBorg build jackmix klick mixxx nova-filters rhvoice giv mypaint swift-im bombono mapnik serf pythonPackages.Nuitka pythonPackages.pyexiv2 godot hammer toluapp btanks dxx-rebirth endless-sky globulation2 the-powder-toy fceux gpsd mongodb mariadb lprof a2jmidid ams-lv2 ardour fomp guitarix ingen jalv mda_lv2 non patchage hamster-time-tracker kupfer xiphos xfce.xfce4-dockbarx-plugin xfce.xfce4-namebar-plugin dropbox clasp aubio lilv lv2 lvtk ntk raul sratom suil ganv ndn-cxx ns-3 serd sord termbox wxmupen64plus jackaudio pflask blockhash glmark2 weighttp stp ingen |
GrahamcOfBorg
added
6.topic: python
6.topic: xfce
|
Unexpected error: unexpected build failure on aarch64-linux (full log) Attempted: jackmix, klick, mixxx, nova-filters, rhvoice, giv, mypaint, swift-im, bombono, mapnik, serf, pythonPackages.Nuitka, hammer, toluapp, btanks, dxx-rebirth, endless-sky, globulation2, fceux, gpsd, mongodb, mariadb, lprof, ams-lv2, ardour, fomp, guitarix, ingen, jalv, mda_lv2, non, patchage, hamster-time-tracker, kupfer, xiphos, xfce.xfce4-dockbarx-plugin, xfce.xfce4-namebar-plugin, clasp, aubio, lilv, lv2, lvtk, ntk, raul, sratom, suil, ganv, ndn-cxx, ns-3, serd, sord, termbox, wxmupen64plus, pflask, blockhash, glmark2, weighttp, stp, ingen The following builds were skipped because they don't evaluate on aarch64-linux: pythonPackages.pyexiv2, godot, the-powder-toy, a2jmidid, dropbox, jackaudio Partial log (click to expand)
|
|
Timed out, unknown build status on x86_64-darwin (full log) Attempted: btanks, mongodb, scons, wafHook The following builds were skipped because they don't evaluate on x86_64-darwin: gpsd, swift-im, swiften, wxmupen64plus Partial log (click to expand)
|
|
Timed out, unknown build status on x86_64-darwin (full log) Attempted: rhvoice, mapnik, serf, pythonPackages.Nuitka, toluapp, btanks, the-powder-toy, mongodb, mariadb, clasp, ndn-cxx, ns-3, termbox, blockhash, weighttp The following builds were skipped because they don't evaluate on x86_64-darwin: jackmix, klick, mixxx, nova-filters, giv, mypaint, swift-im, bombono, pythonPackages.pyexiv2, godot, hammer, dxx-rebirth, endless-sky, globulation2, fceux, gpsd, lprof, a2jmidid, ams-lv2, ardour, fomp, guitarix, ingen, jalv, mda_lv2, non, patchage, hamster-time-tracker, kupfer, xiphos, xfce.xfce4-dockbarx-plugin, xfce.xfce4-namebar-plugin, dropbox, aubio, lilv, lv2, lvtk, ntk, raul, sratom, suil, ganv, serd, sord, wxmupen64plus, jackaudio, pflask, glmark2, stp, ingen Partial log (click to expand)
|
|
I think you missed ns-3 but that's lucky since I've changed the package a bit, I can take care of updating it. I've meant to do something like this for waf but never found the time. Very happy you did it. Waf does break its interface on a regular basis, how easy is it to specify a different waf version ? |
|
Timed out, unknown build status on x86_64-linux (full log) Attempted: btanks, gpsd, mongodb, scons, swift-im, swiften, wafHook, wxmupen64plus Partial log (click to expand)
|
|
Timed out, unknown build status on aarch64-linux (full log) Attempted: btanks, gpsd, mongodb, scons, swift-im, swiften, wafHook, wxmupen64plus Partial log (click to expand)
|
orivej
approved these changes
Nov 13, 2018
| @@ -21983,7 +21978,6 @@ with pkgs; | |||
| dell-530cdn = callPackage ../misc/drivers/dell-530cdn {}; | |||
|
|
|||
| dosbox = callPackage ../misc/emulators/dosbox { }; | |||
| dosbox-unstable = callPackage ../misc/emulators/dosbox/unstable.nix { }; | |||
pkgs/top-level/all-packages.nix
Outdated
| @@ -21533,7 +21529,6 @@ with pkgs; | |||
| mcrl2 = callPackage ../applications/science/logic/mcrl2 { }; | |||
|
|
|||
| minisat = callPackage ../applications/science/logic/minisat {}; | |||
| minisatUnstable = callPackage ../applications/science/logic/minisat/unstable.nix {}; | |||
There was a problem hiding this comment.
No, minisutUnstable is newer than the latest minisat release.
There was a problem hiding this comment.
Ok I can leave this in.
| @@ -18227,8 +18225,6 @@ with pkgs; | |||
|
|
|||
| sxhkd = callPackage ../applications/window-managers/sxhkd { }; | |||
|
|
|||
| sxhkd-unstable = callPackage ../applications/window-managers/sxhkd/unstable.nix { }; | |||
| @@ -15983,8 +15983,6 @@ with pkgs; | |||
|
|
|||
| bspwm = callPackage ../applications/window-managers/bspwm { }; | |||
|
|
|||
| bspwm-unstable = callPackage ../applications/window-managers/bspwm/unstable.nix { }; | |||
pkgs/top-level/all-packages.nix
Outdated
| @@ -12158,7 +12159,6 @@ with pkgs; | |||
| lilv = callPackage ../development/libraries/audio/lilv { }; | |||
|
|
|||
| lv2 = callPackage ../development/libraries/audio/lv2 { }; | |||
| lv2Unstable = callPackage ../development/libraries/audio/lv2/unstable.nix { }; | |||
There was a problem hiding this comment.
Ok I can leave this in.
| @@ -3490,7 +3489,6 @@ with pkgs; | |||
| ispike = callPackage ../development/libraries/science/robotics/ispike { }; | |||
|
|
|||
| isync = callPackage ../tools/networking/isync { }; | |||
| isyncUnstable = callPackage ../tools/networking/isync/unstable.nix { }; | |||
pkgs/top-level/all-packages.nix
Outdated
| @@ -1092,7 +1092,6 @@ with pkgs; | |||
| }; | |||
|
|
|||
| cataract = callPackage ../applications/misc/cataract { }; | |||
| cataract-unstable = callPackage ../applications/misc/cataract/unstable.nix { }; | |||
There was a problem hiding this comment.
No, cataract-unstable is newer.
There was a problem hiding this comment.
Ok I can leave this in.
| fi | ||
|
|
||
| if [ -z "${checkTarget:-}" ]; then | ||
| echo "no check/test target in ${makefile:-Makefile}, doing nothing" |
| if scons check >/dev/null 2>&1; then | ||
| checkTarget=check | ||
| elif scons test >/dev/null 2>&1; then | ||
| checkTarget=test |
There was a problem hiding this comment.
Does this run tests, and assume that they don't exist if they fail?
There was a problem hiding this comment.
Good catch! Just need a -n flag to do a dry run.
| buildFlags="${prefixKey:-prefix=}$prefix $buildFlags" | ||
| fi | ||
|
|
||
| # Old bash empty array hack |
There was a problem hiding this comment.
I don't think the old bash empty array hack should be carried around any more. (You can replace ${sconsFlagsArray+"${sconsFlagsArray[@]}"} with ${sconsFlagsArray[@]}"}.)
|
Timed out, unknown build status on x86_64-linux (full log) Attempted: jackmix, klick, mixxx, nova-filters, rhvoice, giv, mypaint, swift-im, bombono, mapnik, serf, pythonPackages.Nuitka, godot, hammer, toluapp, btanks, dxx-rebirth, endless-sky, globulation2, the-powder-toy, fceux, gpsd, mongodb, mariadb, lprof, a2jmidid, ams-lv2, ardour, fomp, guitarix, ingen, jalv, mda_lv2, non, patchage, hamster-time-tracker, kupfer, xiphos, xfce.xfce4-dockbarx-plugin, xfce.xfce4-namebar-plugin, clasp, aubio, lilv, lv2, lvtk, ntk, raul, sratom, suil, ganv, ndn-cxx, ns-3, serd, sord, termbox, wxmupen64plus, pflask, blockhash, glmark2, weighttp, stp, ingen The following builds were skipped because they don't evaluate on x86_64-linux: pythonPackages.pyexiv2, dropbox, jackaudio Partial log (click to expand)
|
teto
reviewed
Nov 13, 2018
| wafHook = makeSetupHook { | ||
| deps = [ python ]; | ||
| substitutions = { inherit waf; }; | ||
| } ../development/tools/build-managers/waf/setup-hook.sh; |
There was a problem hiding this comment.
should wafHook live in pythonPackages ? or provide waf2Hook for python2 ? Maybe it's ok if you had no problem so far but I know at least one waf-based program (outside nixpkgs) that only supports python2
There was a problem hiding this comment.
waf is usually vendored so I would think it would always support python2? Not 100% sure but I had no problems doing it this way here.
|
Timed out, unknown build status on x86_64-darwin (full log) Attempted: btanks, mongodb, scons, wafHook The following builds were skipped because they don't evaluate on x86_64-darwin: gpsd, swift-im, swiften, wxmupen64plus Partial log (click to expand)
|
|
@GrahamcOfBorg build jackmix klick mixxx nova-filters rhvoice giv mypaint swift-im bombono mapnik serf pythonPackages.Nuitka pythonPackages.pyexiv2 godot hammer toluapp btanks dxx-rebirth endless-sky globulation2 the-powder-toy fceux gpsd mongodb mariadb lprof a2jmidid ams-lv2 ardour fomp guitarix ingen jalv mda_lv2 non patchage hamster-time-tracker kupfer xiphos xfce.xfce4-dockbarx-plugin xfce.xfce4-namebar-plugin dropbox clasp aubio lilv lv2 lvtk ntk raul sratom suil ganv ndn-cxx ns-3 serd sord termbox wxmupen64plus jackaudio pflask blockhash glmark2 weighttp stp ingen |
|
Unexpected error: unexpected build failure on aarch64-linux (full log) Attempted: jackmix, klick, mixxx, nova-filters, rhvoice, giv, mypaint, swift-im, bombono, mapnik, serf, pythonPackages.Nuitka, hammer, toluapp, btanks, dxx-rebirth, endless-sky, globulation2, fceux, gpsd, mongodb, mariadb, lprof, ams-lv2, ardour, fomp, guitarix, ingen, jalv, mda_lv2, non, patchage, hamster-time-tracker, kupfer, xiphos, xfce.xfce4-dockbarx-plugin, xfce.xfce4-namebar-plugin, clasp, aubio, lilv, lv2, lvtk, ntk, raul, sratom, suil, ganv, ndn-cxx, ns-3, serd, sord, termbox, wxmupen64plus, pflask, blockhash, glmark2, weighttp, stp, ingen The following builds were skipped because they don't evaluate on aarch64-linux: pythonPackages.pyexiv2, godot, the-powder-toy, a2jmidid, dropbox, jackaudio Partial log (click to expand)
|
|
Timed out, unknown build status on aarch64-linux (full log) Attempted: btanks, gpsd, mongodb, scons, swift-im, swiften, wafHook, wxmupen64plus Partial log (click to expand)
|
|
Timed out, unknown build status on x86_64-linux (full log) Attempted: btanks, gpsd, mongodb, scons, swift-im, swiften, wafHook, wxmupen64plus Partial log (click to expand)
|
The waf build system is python-based and hosted locally in each package in the executable file named "waf". Unlike CMake, it cannot generate makefiles so we end up having to override the configure, build, and install phases. I've tried to keep these as close to what's in setup.sh as possible. If there is no waf file in the root directory, then we just copy the one hosted in Nixpkgs. Otherwise the only thing you have to add to a package using Waf is "wafHook" into nativeBuildInputs. wafFlags controls the flags specifically passed to waf while configureFlags, buildFlags, and installFlags are still used as in the generic builder.
Replace "waf" phases with wafHook that manages everything automatically. Should make things more modular. Packages affected here are: - a2jmidid - ams-lv2 - ardour - fomp - guitarix - ingen - jalv - mda-lv2 - non - patchage - hamster-time-tracker - kupfer - xiphos - xfce4-dockbarx-plugin - xfce4-namebar-plugin - dropbox - clasp - aubio - liliv - lv2 - lvtk - ntk - raul - sratom - suil - ganv - ndn-cxx - ns3 - serd - sord - termbox - wxmupen64plus - jackaudio - pflask - blockhash - glmark2 - weighttp
The scons build system is python-based and has a binary named scons. Unlike CMake, it cannot generate makefiles so we end up having to override the build, install, and check phases. I have added the setupHook to the scons package so that integration requires no unique steps - just putting scons in nativeBuildInputs should be enough. sconsFlags controls the flags specifically passed to scons while buildFlags, installFlags, and checkFlags should still be usable. Some packages use different names for the prefix flag. In those cases you will have to set "prefixKey" to something like "PREFIX=" as there are multiple names for the "prefix" used in scons.
Lots of packages can use it. Here is the list: - jackmix - klick - mixx - nova-filters - rhvoice - giv - mypaint - swift-im - bombono - mapnik - serf - nuitka - pyexiv2 - godot - hammer - toluapp - btanks - dxx-rebirth - endless-sky - globulation - the-powder-toy - fceux - gpsd - mongodb - rippled - mariadb - lprof
There’s nothing wrong with unstable packages when they are maintained and frequently updated. However, when they become out-of-date, as many do, it is usually best to just get rid of them as the stable version is become newer than the unstable version. This removes any packages called "unstable" that have not been updated in over 1 year. Affected packages include: - isyncUnstable - sxhkd-unstable - dosbox-unstable Revert "treewide: remove unstable packages" This reverts commit df01b0b.
This flag seems to make things work again.
matthewbauer
force-pushed
the
cleanups2
branch
from
a1fe4de
to
83d02fa
Compare
|
Timed out, unknown build status on x86_64-darwin (full log) Attempted: rhvoice, mapnik, serf, pythonPackages.Nuitka, toluapp, btanks, the-powder-toy, mongodb, mariadb, clasp, ndn-cxx, ns-3, termbox, blockhash, weighttp The following builds were skipped because they don't evaluate on x86_64-darwin: jackmix, klick, mixxx, nova-filters, giv, mypaint, swift-im, bombono, pythonPackages.pyexiv2, godot, hammer, dxx-rebirth, endless-sky, globulation2, fceux, gpsd, lprof, a2jmidid, ams-lv2, ardour, fomp, guitarix, ingen, jalv, mda_lv2, non, patchage, hamster-time-tracker, kupfer, xiphos, xfce.xfce4-dockbarx-plugin, xfce.xfce4-namebar-plugin, dropbox, aubio, lilv, lv2, lvtk, ntk, raul, sratom, suil, ganv, serd, sord, wxmupen64plus, jackaudio, pflask, glmark2, stp, ingen Partial log (click to expand)
|
|
Success on x86_64-linux (full log) Attempted: scons, wafHook, wxmupen64plus Partial log (click to expand)
|
|
Success on x86_64-darwin (full log) Attempted: scons, wafHook The following builds were skipped because they don't evaluate on x86_64-darwin: wxmupen64plus Partial log (click to expand)
|
|
Timed out, unknown build status on x86_64-linux (full log) Attempted: btanks, gpsd, mongodb, scons, swift-im, swiften, wafHook, wxmupen64plus Partial log (click to expand)
|
|
Failure on aarch64-linux (full log) Attempted: scons, wafHook, wxmupen64plus Partial log (click to expand)
|
|
Timed out, unknown build status on x86_64-darwin (full log) Attempted: btanks, mongodb, scons, wafHook The following builds were skipped because they don't evaluate on x86_64-darwin: gpsd, swift-im, swiften, wxmupen64plus Partial log (click to expand)
|
|
Timed out, unknown build status on x86_64-linux (full log) Attempted: jackmix, klick, mixxx, nova-filters, rhvoice, giv, mypaint, swift-im, bombono, mapnik, serf, pythonPackages.Nuitka, godot, hammer, toluapp, btanks, dxx-rebirth, endless-sky, globulation2, the-powder-toy, fceux, gpsd, mongodb, mariadb, lprof, a2jmidid, ams-lv2, ardour, fomp, guitarix, ingen, jalv, mda_lv2, non, patchage, hamster-time-tracker, kupfer, xiphos, xfce.xfce4-dockbarx-plugin, xfce.xfce4-namebar-plugin, clasp, aubio, lilv, lv2, lvtk, ntk, raul, sratom, suil, ganv, ndn-cxx, ns-3, serd, sord, termbox, wxmupen64plus, pflask, blockhash, glmark2, weighttp, stp, ingen The following builds were skipped because they don't evaluate on x86_64-linux: pythonPackages.pyexiv2, dropbox, jackaudio Partial log (click to expand)
|
|
Timed out, unknown build status on aarch64-linux (full log) Attempted: btanks, gpsd, mongodb, scons, swift-im, swiften, wafHook, wxmupen64plus Partial log (click to expand)
|
matthewbauer
added a commit
that referenced
this pull request
Nov 14, 2018
Fallout fixes from #50293
bhipple
reviewed
Dec 5, 2018
|
|
||
| if ! [ -f ./waf ]; then | ||
| cp @waf@ waf | ||
| fi |
There was a problem hiding this comment.
Out of curiosity, why did you elect to use cp instead of ln?
There was a problem hiding this comment.
I can't remember tbh. ln might have been a better idea but I think I had a package that was already doing this in configurePhase that used cp.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds some setup hooks to use in scons and waf build systems. This makes things more modular and easier to make changes with. I think I rewrote everything to use them but I could have missed some. I would appreciate any tips on missing packages. One of the biggest benefits of this, is that it should enable parallel building for a lot of packages that were not specifically passing NIX_BUILD_CORES previously. In addition, it should split the phases correctly. Overall we end up getting rid of about 400 lines of code with this change.
This is kind of a big PR. I am hoping it will be a good change, though, and I've left it as a mass diff. If requested, I can split it up into a few different PRs though.
Waf
The waf build system is python-based and hosted locally in each package in the executable file named "waf". Unlike CMake, it cannot generate makefiles so we end up having to override the configure, build, and install phases. I've tried to keep these as close to what's in setup.sh as possible. If there is no waf file in the root directory, then we just copy the one hosted in Nixpkgs. Otherwise the only thing you have to add to a package using Waf is "wafHook" into nativeBuildInputs. wafFlags controls the flags specifically passed to waf while configureFlags, buildFlags, and installFlags are still used as in the generic builder.
Scons
The scons build system is python-based and has a binary named scons. Unlike CMake, it cannot generate makefiles so we end up having to override the build, install, and check phases. I have added the setupHook to the scons package so that integration requires no unique steps - just putting scons in nativeBuildInputs should be enough. sconsFlags controls the flags specifically passed to scons while buildFlags, installFlags, and checkFlags should still be usable. Some packages use different names for the prefix flag. In those cases you will have to set "prefixKey" to something like "PREFIX=" as there are multiple names for the "prefix" used in scons.
Changed packages
Package affected by this due to using either Scons or Waf:
jackmix klick mixxx nova-filters rhvoice giv mypaint swift-im bombono mapnik serf pythonPackages.Nuitka pythonPackages.pyexiv2 godot hammer toluapp btanks dxx-rebirth endless-sky globulation2 the-powder-toy fceux gpsd mongodb mariadb lprof a2jmidid ams-lv2 ardour fomp guitarix ingen jalv mda_lv2 non patchage hamster-time-tracker kupfer xiphos xfce.xfce4-dockbarx-plugin xfce.xfce4-namebar-plugin dropbox clasp aubio lilv lv2 lvtk ntk raul sratom suil ganv ndn-cxx ns-3 serd sord termbox wxmupen64plus jackaudio pflask blockhash glmark2 weighttp stp ingen
Removed packages
Some old "unstable" packages have gone outdated. I've gone though any package older than 1 year and removed it from Nixpkgs. The stable version can still be used. Here is the list of packages: