-
-
Notifications
You must be signed in to change notification settings - Fork 15.5k
nodePackages: regenerate with node2nix 1.9.0 #110813
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
I still need to investigate the problem case as described in #110545, most likely caused by |
Sorry dear bot @ofborg, I'm following this pull request with interest, but I am unfortunately not qualified to contribute a review. |
It is definitely zigbee2mqtt because this exact same problem happened once before. When someone tries this with nixpkgs-review make sure to pass |
hmm the weird thing is, I just have successfully built Also, the following evaluation: nix-env -f . -qaP --xml --out-path --show-trace works for me. Hmm indeed I get a stack stackoverflow/infinite recursion when I run: nix-review wip |
You need to run |
@SuperSandro2000 yes I know. This is weird situation. I think this problem is not caused by infinite recursion, but most likely by recursing too deeply. Maybe this condition is triggered when we evaluate all derivations on a cold cache, but not an individual package or a sub set. What I think might contribute to this problem is the store derivations of NPM package deployments -- basically all dependencies including transitive dependencies in the entire closure become a direct dependency of the package to be deployed. Sadly, I see more and more NPM projects that use a substantial amount of dependencies (mostly micro-packages with only single functions). For example, I see many packages in the closure of The only way to break out of this cycle, is probably a partial upgrade in which some packages are deployed with node2nix 1.9.0 and some still with 1.8.0. A binary cache probably contributes to the fact that not everything needs to be evaluated. The real solution, of course, would be to change |
I almost hate myself for saying that but I think maybe it doesn't make sense to block all of this work on one package. It's easy for me to say that since I don't depend on it and neither I knew the tool prior to this discussion but..... I think we should at least consider the option of just simply marking it as a broken. |
We can keep zigbee2mqtt on an old version for now or worst case we mark it broken. |
I'll see if I can work out a gradual transition scenario |
40fc29c
to
8ba7dfa
Compare
@SuperSandro2000 I have regenerated the entire package set against a more recent version of the master branch. Moreover, I have left @turboMaCk can you also take another look at this PR? Since this is an high impact change, I could use a second pair of eyes. @raboof Maybe you can check as well, since previously you have noticed the issue. |
I don't see it with this PR anymore, either! |
So I've ran There seems to be few problems still.
For elmPackages I'm aware of the breaking changes in upstream from #110845. I would suggest to reset/revert the changes - I'll take care of them in separate PR. I also need to sync with upstreams on a few things. I'm not sure about the rest though. Let me know if I could help you with this further. |
8ba7dfa
to
e752616
Compare
@turboMaCk I just fixed the About reverting changes -- do you mean reverting the changes that are related to the Elm packages? |
e752616
to
cb52887
Compare
hmm it closed all of a sudden :( |
be2a98a
to
9c6cafc
Compare
@turboMaCk I reverted the |
@svanderburg I've actually tested it locally before posting the reply and I'll also let nixpkgs-review ran again (this time even my old macbook pro to see how this does on Darwin) probably over the night if it wil take too long. But I think this is 👍 from my side at least. |
@turboMaCk I also briefly checked the other two packages that did not build, but they fail because some tool (that has an NPM package as a dependency) in the build process fails, with a segfault. Although I can't be 100% sure, I think it's very unlikely that it was caused by an NPM dependency upgrade. It looks like |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the result of nixpkgs-review on nixos:
nixpkgs-review pr 110813 --eval local
$ git -c fetch.prune=false fetch --force https://github.com/NixOS/nixpkgs master:refs/nixpkgs-review/0 pull/110813/head:refs/nixpkgs-review/1
remote: Enumerating objects: 157, done.
remote: Counting objects: 100% (157/157), done.
remote: Compressing objects: 100% (18/18), done.
remote: Total 205 (delta 142), reused 147 (delta 137), pack-reused 48
Receiving objects: 100% (205/205), 143.80 KiB | 871.00 KiB/s, done.
Resolving deltas: 100% (143/143), completed with 91 local objects.
From https://github.com/NixOS/nixpkgs
410bbd828cd..c0974f36049 master -> refs/nixpkgs-review/0
+ 8ba7dfad5cb...9c6cafc820c refs/pull/110813/head -> refs/nixpkgs-review/1 (forced update)
$ git worktree add /home/marek/.cache/nixpkgs-review/pr-110813-4/nixpkgs c0974f3604980e548607ad2587d267b0a74bb932
Preparing worktree (detached HEAD c0974f36049)
Updating files: 100% (24297/24297), done.
HEAD is now at c0974f36049 Merge pull request #111669 from apeyroux/up-masterpdf-5720
$ nix-env -f /home/marek/.cache/nixpkgs-review/pr-110813-4/nixpkgs -qaP --xml --out-path --show-trace
$ git merge --no-commit 9c6cafc820c547107a33c3f9057c08033c30be36
Removing pkgs/servers/matrix-synapse/matrix-appservice-slack/package.json
Removing pkgs/applications/networking/cluster/spacegun/package.json
Automatic merge went well; stopped before committing as requested
$ nix-env -f /home/marek/.cache/nixpkgs-review/pr-110813-4/nixpkgs -qaP --xml --out-path --show-trace --meta
124 packages updated:
airfield bat-extras-prettybat bitwarden-cli emojione EPGStation etcher imgbrd-grabber iosevka ldgallery lumo nixui node_base16-builder node_castnow node_create-cycle-app node_cryptpad node_dat node_fast-cli flood (4.3.1 → 4.4.0) node_fx node_gtop node_hueadm node_imapnotify node_joplin lessc (4.1.0 → 4.1.1) node_mastodon-bot node_matrix-appservice-slack node_mirakurun n8n (0.96.0 → 0.105.0) netlify-cli (3.4.3 → 3.4.6) newman (5.2.1 → 5.2.2) pm2 (4.5.1 → 4.5.4) node_postcss-cli node_pscid node_pulp node_purescript-psa redoc-cli (0.10.2 → 0.10.3) node_shout node_sloc node_spacegun node_thelounge node_triton node_whitebophir node_wring node_yaml-language-server node__at_antora_slash_cli node__at_google_slash_clasp node__at_webassemblyjs_slash_cli node__at_webassemblyjs_slash_repl node__at_webassemblyjs_slash_wasm-strip node__at_webassemblyjs_slash_wasm-text-gen node__at_webassemblyjs_slash_wast-refmt parity-ui python3.7-batchspawner python3.7-dockerspawner python3.7-jupyterhub python3.7-jupyterhub-ldapauthenticator python3.7-jupyterhub-systemdspawner python3.7-jupyterhub-tmpauthenticator python3.7-oauthenticator python3.8-batchspawner python3.8-dockerspawner python3.8-jupyterhub python3.8-jupyterhub-ldapauthenticator python3.8-jupyterhub-systemdspawner python3.8-jupyterhub-tmpauthenticator python3.8-oauthenticator python3.9-batchspawner python3.9-dockerspawner python3.9-jupyterhub python3.9-jupyterhub-ldapauthenticator python3.9-jupyterhub-systemdspawner python3.9-jupyterhub-tmpauthenticator python3.9-oauthenticator remarkjs ripcord sage sage sage slack slack styx teleprompter twemoji-color-font uchiwa vimplugin-coc-css vimplugin-coc-diagnostic vimplugin-coc-emmet vimplugin-coc-eslint vimplugin-coc-git vimplugin-coc-go vimplugin-coc-highlight vimplugin-coc-html vimplugin-coc-imselect vimplugin-coc-java vimplugin-coc-jest vimplugin-coc-json vimplugin-coc-lists vimplugin-coc-markdownlint vimplugin-coc-metals vimplugin-coc-pairs vimplugin-coc-prettier vimplugin-coc-python vimplugin-coc-r-lsp vimplugin-coc-rls vimPlugins.coc-rust-analyzer (0.30.0 → 0.31.0) vimplugin-coc-smartf vimplugin-coc-snippets vimplugin-coc-solargraph vimplugin-coc-stylelint vimplugin-coc-tabnine vimplugin-coc-tslint vimplugin-coc-tslint-plugin vimplugin-coc-tsserver vimplugin-coc-vetur vimplugin-coc-vimlsp vimplugin-coc-vimtex vimplugin-coc-wxml vimplugin-coc-yaml vimplugin-coc-yank vimplugin-YouCompleteMe vscode-extension-MS-python-vscode-pylance vscode-extension-rust-analyzer-unstable vscode-extension-vscode-lldb ycmd
$ nix --experimental-features nix-command build --no-link --keep-going --option build-use-sandbox relaxed -f /home/marek/.cache/nixpkgs-review/pr-110813-4/build.nix
builder for '/nix/store/nz24mhs18vadzw1q6ljzvxxhcd3lmhmi-vscode-lldb-adapter-1.5.3.drv' failed with exit code 101; last 10 log lines:
For more information about this error, try `rustc --explain E0277`.
The following warnings were emitted during compilation:
warning: Could not find CMakeCache.txt
error: could not compile `lldb`
To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error: build failed
cannot build derivation '/nix/store/wlfk2q8q2l9kb56dyf955ygjbsckgmfh-vscode-lldb-1.5.3-vsix.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/x9wyv5bvg01zzybz8wfwpl5plx11pvx0-vscode-extension-vscode-lldb.drv': 1 dependencies couldn't be built
builder for '/nix/store/xm4x8gx5nzisig0ablf1a17hz7gka9p7-sage-tests-9.2.drv' failed with exit code 1; last 10 log lines:
[13 tests, 352.44 s]
----------------------------------------------------------------------
sage -t --long --random-seed=0 /nix/store/kc7cf6p4z8qm7hppw40kazlwyv54p2fr-sage-src-9.2/src/sage/plot/plot3d/base.pyx # 11 doctests failed
sage -t --long --random-seed=0 /nix/store/kc7cf6p4z8qm7hppw40kazlwyv54p2fr-sage-src-9.2/src/sage/plot/animate.py # 2 doctests failed
sage -t --long --random-seed=0 /nix/store/kc7cf6p4z8qm7hppw40kazlwyv54p2fr-sage-src-9.2/src/sage/repl/rich_output/backend_ipython.py # 1 doctest failed
sage -t --long --random-seed=0 /nix/store/kc7cf6p4z8qm7hppw40kazlwyv54p2fr-sage-src-9.2/src/sage/repl/rich_output/display_manager.py # 1 doctest failed
----------------------------------------------------------------------
Total time for all tests: 1605.9 seconds
cpu time: 17281.0 seconds
cumulative wall time: 19924.2 seconds
cannot build derivation '/nix/store/nw6afad4d7nhwhrxrs4r23dwsm5h40p8-sage-9.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/x8079n1y6b9awabwiqafqirfaqkv879s-sage-9.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/m5i1px9c29c7cgaxnkkgdvxsah4qvawk-env.drv': 3 dependencies couldn't be built
error: --- Error ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
build of '/nix/store/m5i1px9c29c7cgaxnkkgdvxsah4qvawk-env.drv' failed
Link to currently reviewing PR:
https://github.com/NixOS/nixpkgs/pull/110813
3 packages marked as broken and skipped:
dat remarkjs vimPlugins.coc-imselect
3 packages failed to build:
sage sageWithDoc vscode-extensions.vadimcn.vscode-lldb
116 packages built:
airfield antora base16-builder bat-extras.prettybat bitwarden-cli castnow create-cycle-app cryptpad emojione epgstation etcher fast-cli flood fx google-clasp gtop hueadm imapnotify imgbrd-grabber iosevka joplin ldgallery lessc lumo mastodon-bot matrix-appservice-slack mirakurun n8n netlify-cli newman nixui parity-ui pm2 postcss-cli pscid pulp purescript-psa python37Packages.batchspawner python37Packages.dockerspawner python37Packages.jupyterhub python37Packages.jupyterhub-ldapauthenticator python37Packages.jupyterhub-systemdspawner python37Packages.jupyterhub-tmpauthenticator python37Packages.oauthenticator python38Packages.batchspawner python38Packages.dockerspawner python38Packages.jupyterhub python38Packages.jupyterhub-ldapauthenticator python38Packages.jupyterhub-systemdspawner python38Packages.jupyterhub-tmpauthenticator python38Packages.oauthenticator python39Packages.batchspawner python39Packages.dockerspawner python39Packages.jupyterhub python39Packages.jupyterhub-ldapauthenticator python39Packages.jupyterhub-systemdspawner python39Packages.jupyterhub-tmpauthenticator python39Packages.oauthenticator redoc-cli ripcord shout slack sloc spacegun styx teleprompter thelounge triton twemoji-color-font uchiwa vimPlugins.YouCompleteMe vimPlugins.coc-css vimPlugins.coc-diagnostic vimPlugins.coc-emmet vimPlugins.coc-eslint vimPlugins.coc-git vimPlugins.coc-go vimPlugins.coc-highlight vimPlugins.coc-html vimPlugins.coc-java vimPlugins.coc-jest vimPlugins.coc-json vimPlugins.coc-lists vimPlugins.coc-markdownlint vimPlugins.coc-metals vimPlugins.coc-pairs vimPlugins.coc-prettier vimPlugins.coc-python vimPlugins.coc-r-lsp vimPlugins.coc-rls vimPlugins.coc-rust-analyzer vimPlugins.coc-smartf vimPlugins.coc-snippets vimPlugins.coc-solargraph vimPlugins.coc-stylelint vimPlugins.coc-tabnine vimPlugins.coc-tslint vimPlugins.coc-tslint-plugin vimPlugins.coc-tsserver vimPlugins.coc-vetur vimPlugins.coc-vimlsp vimPlugins.coc-vimtex vimPlugins.coc-wxml vimPlugins.coc-yaml vimPlugins.coc-yank vscode-extensions.matklad.rust-analyzer vscode-extensions.ms-python.vscode-pylance wasm-strip wasm-text-gen wast-refmt webassemblyjs-cli webassemblyjs-repl whitebophir wring yaml-language-server ycmd
error: --- Error ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
build log of '/nix/store/9rggvly206hpc9b9k4h3qp0xj18lfl7k-lumo-1.10.1.drv' is not available
error: --- Error ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
build log of '/nix/store/nw6afad4d7nhwhrxrs4r23dwsm5h40p8-sage-9.2.drv' is not available
error: --- Error ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
build log of '/nix/store/x8079n1y6b9awabwiqafqirfaqkv879s-sage-9.2.drv' is not available
error: --- Error ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
build log of '/nix/store/x9wyv5bvg01zzybz8wfwpl5plx11pvx0-vscode-extension-vscode-lldb.drv' is not available
$ nix-shell /home/marek/.cache/nixpkgs-review/pr-110813-4/shell.nix
I also think these are most likely unrelated failures. MacOS build is still running since it's much weaker hw. So far the only unexpected thing is failure of ldgallery-viewer
there but I think that's most likely not a regression.
Overall I think this PR is more than good enough considering the scope. I really appreciate the effort! From my side it's LGTM.
@turboMaCk Thank you for all the assistance! I believe in the previous PR (that we reverted) we came to the same conclusion that the I'll wait until tomorrow, so that people still have the opportunity to respond. Hopefully, nothing in the I also have to give Because NPM works so well, people are installing more and more package dependencies in projects (and those packages do the same) causing explosions of dependencies. Probably in the near future, we may run into more weird limitations that the Nix package manager/Nix expression languages has difficulties coping with. That's why I think we have to be careful with adding new NPM projects to Nixpkgs, until I found a better to deal with these big/complex projects. This is still ongoing research. |
@ofborg eval |
This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch). Result of 1 package marked as broken and skipped:
1 package failed to build and already failed to build on hydra master:
1 package failed to build and are new build failure:
28 packages built:
The following issues got detected with the above build packages. python37Packages.jupyterhub-systemdspawner: Zero tests run by pytest got detected: 'Ran 0 tests in 0.000s'
Please consider this feature to be alpha. A substituteInPlace with an unmatched pattern got detected:
Please check the offending substituteInPlace for typos or changes in source. Zero tests run by pytest got detected: 'Ran 0 tests in 0.000s'
Please consider this feature to be alpha. A substituteInPlace with an unmatched pattern got detected:
Please check the offending substituteInPlace for typos or changes in source. Zero tests run by pytest got detected: 'Ran 0 tests in 0.000s'
Please consider this feature to be alpha. A substituteInPlace with an unmatched pattern got detected:
Please check the offending substituteInPlace for typos or changes in source. Please consider this feature to be alpha. A substituteInPlace with an unmatched pattern got detected:
Please check the offending substituteInPlace for typos or changes in source. Near pkgs/development/tools/misc/ycmd/default.nix:26:3:
See: https://github.com/jtojnar/nixpkgs-hammering/blob/master/explanations/missing-phase-hooks.md Near pkgs/development/tools/misc/ycmd/default.nix:43:3:
See: https://github.com/jtojnar/nixpkgs-hammering/blob/master/explanations/missing-phase-hooks.md Near pkgs/development/tools/misc/ycmd/default.nix:96:5:
See: https://github.com/jtojnar/nixpkgs-hammering/blob/master/explanations/unclear-gpl.md |
@SuperSandro2000 Looks like the output matches our expectation. The only exception is the failing The rest (I believe) should not cause any problems. |
I have blacklisted writers test in nixpkgs-review. The other things are expected. |
This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch). Result of 3 packages marked as broken and skipped:
1 package blacklisted:
1 package failed to build and are new build failure:
114 packages built:
The following issues got detected with the above build packages. python37Packages.jupyterhub: Please consider this feature to be alpha. A substituteInPlace with an unmatched pattern got detected:
Please check the offending substituteInPlace for typos or changes in source. Zero tests run by pytest got detected: 'Ran 0 tests in 0.000s'
Please consider this feature to be alpha. A substituteInPlace with an unmatched pattern got detected:
Please check the offending substituteInPlace for typos or changes in source. Zero tests run by pytest got detected: 'Ran 0 tests in 0.000s'
Please consider this feature to be alpha. A substituteInPlace with an unmatched pattern got detected:
Please check the offending substituteInPlace for typos or changes in source. Zero tests run by pytest got detected: 'Ran 0 tests in 0.000s'
Please consider this feature to be alpha. A substituteInPlace with an unmatched pattern got detected:
Please check the offending substituteInPlace for typos or changes in source. Near pkgs/development/tools/misc/ycmd/default.nix:96:5:
See: https://github.com/jtojnar/nixpkgs-hammering/blob/master/explanations/unclear-gpl.md Near pkgs/development/tools/misc/ycmd/default.nix:26:3:
See: https://github.com/jtojnar/nixpkgs-hammering/blob/master/explanations/missing-phase-hooks.md Near pkgs/development/tools/misc/ycmd/default.nix:43:3:
See: https://github.com/jtojnar/nixpkgs-hammering/blob/master/explanations/missing-phase-hooks.md |
pkgs/stdenv/generic/default.nix
Outdated
@@ -153,7 +153,7 @@ let | |||
}) mkDerivation; | |||
|
|||
# Slated for deprecation in 21.11 | |||
lib = builtins.trace | |||
lib = throw #builtins.trace |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This needs to be reverted.
9c6cafc
to
84a0978
Compare
@SuperSandro2000 I just reverted that change (I did an amend/force push to make sure it won't get into We also briefly encountered that problem with the |
Anyway, I think we have pretty much everything covered now. I'm now awaiting the new evaluation evaluation status. |
@SuperSandro2000 @turboMaCk I think we're ready right? I intend to merge this very soon, or feel free to click on the merge button 👍 |
Agree - 👍 merge |
@timokau @omasanori I forgot to disable the intermittent segfaulting Sage test, which masked the fact that a three.js upgrade here broke another Sage test. Apologies for not taking action earlier, I will fix the three.js breakage and disable the other test today. |
This reverts commit 3cd3145.
Motivation for this change
Second attempt update to node2nix 1.9.0.
Things done
sandbox
innix.conf
on non-NixOS linux)nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
./result/bin/
)nix path-info -S
before and after)