nodePackages: Switch to 12.x and 14.x, continue #85764 #89184

calbrecht · 2020-05-29T20:17:09Z

Motivation for this change

To continue with PR #85764 and implement suggested changes on top of that.

To be consistent with future updates of Node.js, i propose (and implement within this PR) on top of the suggested changes in mentionend PR, that we stick to the official released Node.js versions LTS and Current, now at 12.x and 14.x.

Let us reflect that in the change of the nodePackages attribute in the top-level/all-packages.nix package set by renaming the version suffixed package set to nodePackages_latest for the latest official Current release, and nodePackages for the latest official LTS release.

Things done

To ensure all node-packages build successfully, i wrote a shell.nix for testing.

shell.nix in nixpkgs before PR (hacky, but you get the point)

{ pkgs ? import ./default.nix {} }:

let
  node10Pkgs = (import ./pkgs/development/node-packages/default-v10.nix {
    inherit pkgs;
    inherit (pkgs) stdenv;
    nodejs = pkgs.nodejs;
  });
  node12Pkgs = (import ./pkgs/development/node-packages/default-v12.nix {
    inherit pkgs;
    inherit (pkgs) stdenv;
    nodejs = pkgs.nodejs-12_x;
  });
  node13Pkgs = (import ./pkgs/development/node-packages/default-v13.nix {
    inherit pkgs;
    inherit (pkgs) stdenv;
    nodejs = pkgs.nodejs-13_x;
  });
in
pkgs.mkShell {

  buildInputs = with pkgs.lib;
  (attrValues (
    filterAttrs (name: _: 1 > length (
      intersectLists [ name ] [
        # broken packages 10_x
        "coc-imselect"
        "fast-cli-1.x"
        "git-ssb"
        "pulp"
        "insect"
        "node-red"
        "node-inspector"
        "stf"
        "ssb-server"
        # not applicable 10_x
        "ios-deploy"
      ])) node10Pkgs
  )) ++
  (attrValues (
    filterAttrs (name: _: 1 > length (
      intersectLists [ name ] [
        # broken packages 12_x
        # not applicable 12_x
      ])) node12Pkgs
  )) ++
  (attrValues (
    filterAttrs (name: _: 1 > length (
      intersectLists [ name ] [
        # broken packages 13_x
        # not applicable 13_x
      ])) node13Pkgs
  ));

  shellHook = ''
    ${pkgs.lib.concatMapStrings (name: ''
      echo -n 10: ${name}:\ ; echo -en $(which ${name} 2>/dev/null)"\n" ;
    '') (pkgs.lib.attrNames node10Pkgs) }
    ${pkgs.lib.concatMapStrings (name: ''
      echo -n 12: ${name}:\ ; echo -en $(which ${name} 2>/dev/null)"\n" ;
    '') (pkgs.lib.attrNames node12Pkgs) }
    ${pkgs.lib.concatMapStrings (name: ''
      echo -n 13: ${name}:\ ; echo -en $(which ${name} 2>/dev/null)"\n" ;
    '') (pkgs.lib.attrNames node13Pkgs) }
  '';
}

With the help of this shell.nix i noticed there are some packages already broken in the old node-packages. Namely coc-imselect fast-cli-1.x git-ssb pulp insect node-red node-inspector stf ssb-server and one package that claims to be incompatible with linux, namely ios-deploy.

So i thought about what to do if node-packages break with new releases and in different versions, and i came to the conclusion the easiest way is to mark them as broken in the meta attribute depending on the nodejs.version in use. Similar method is applicable to platform support with the meta.platforms attribute. So we are able to write in node-packages/default.nix e.g.

let
  since = (version: pkgs.lib.versionAtLeast nodejs.version version);
in nodePackages // {
  # ....
  "fast-cli-1.x" = nodePackages."fast-cli-1.x".override {
    meta.broken = since "10";
  };
}

To ensure that is working i came up with this shell.nix

shell.nix in nixpkgs after PR

{ pkgs ? import ./default.nix {} }:

let
  isBroken = pkg: pkgs.lib.hasAttrByPath ["meta" "broken"] pkg && pkg.meta.broken;
  isAvailable = pkg: pkgs.lib.hasAttrByPath ["meta" "available"] pkg && pkg.meta.available;
  toInputs = nodePkgs: pkgs.lib.mapAttrsToList (_: pkg: pkg) nodePkgs;
  filterPkgs = nodePkgs: pkgs.lib.filterAttrs (_: pkg: ! isBroken pkg && isAvailable pkg) nodePkgs;
  toFilteredInputs = nodePkgs: toInputs (filterPkgs nodePkgs);

  node12Pkgs = pkgs.lib.filterAttrs (_: v: pkgs.lib.isDerivation v) pkgs.nodePackages;
  node14Pkgs = pkgs.lib.filterAttrs (_: v: pkgs.lib.isDerivation v) pkgs.nodePackages_latest;
in
pkgs.mkShell {

  buildInputs = (toFilteredInputs node12Pkgs) ++ (toFilteredInputs node14Pkgs);

  shellHook = ''
    ${pkgs.lib.concatMapStrings (name: ''
      echo -n 12: ${name}:\ ; echo -en $(which ${name} 2>/dev/null)"\n" ;
    '') (pkgs.lib.attrNames node12Pkgs) }
    ${pkgs.lib.concatMapStrings (name: ''
      echo -n 14: ${name}:\ ; echo -en $(which ${name} 2>/dev/null)"\n" ;
    '') (pkgs.lib.attrNames node14Pkgs) }
  '';
}

🎉

cc @Mic92 @marsam @prusnak

The broken node-packages need to be fixed, but i do not consider the fix being part of this PR.

Signed-off-by: Christian Albrecht <christian.albrecht@mayflower.de>

currently LTS 12.x and Current 14.x

pkgs/development/node-packages/node-packages.json

Mic92

Looks good to me, if those missing packages gets re-added.

calbrecht · 2020-05-30T08:07:06Z

Opened #89205 to track/remove/fix broken packages.

prusnak · 2020-05-30T09:59:58Z

Love it! 👍

Since we are regenerating node-packages.nix, can we also change hashes from sha1 to sha256? This would fix most of the problems from #77238

Mic92 · 2020-05-30T10:33:29Z

Those are coming from npm and are not computed locally. Hence we have little control over that. However they might be an indicator for old packages, which we might want to remove.

calbrecht · 2020-05-30T10:37:09Z

@prusnak, yes @Mic92 is right, that would be something that has to be tackled in node2nix with, as there is already some abstraction over generating/converting sha sums.

Also i did the following to check the status:

tutnix ~/ws/nixpkgs-upstream (git) 6149df8fd095 U continue-pull-85764
% rg -o -c 'sha1' pkgs/development/node-packages/node-packages.nix
2342
tutnix ~/ws/nixpkgs-upstream (git) 6149df8fd095 U continue-pull-85764
% rg -o -c 'sha256' pkgs/development/node-packages/node-packages.nix
18
tutnix ~/ws/nixpkgs-upstream (git) 6149df8fd095 U continue-pull-85764
% rg -o -c 'sha512' pkgs/development/node-packages/node-packages.nix
3399

Mic92 · 2020-05-30T10:46:45Z

Also the node ecosystem has bigger problem than sha1 collisions, i.e. malware.

calbrecht · 2020-05-30T11:22:49Z

Also the node ecosystem has bigger problem than sha1 collisions, i.e. malware.

Indeed, but i guess that holds true for any language specific package repository e.g. python

Mic92 · 2020-05-30T11:34:48Z

True also there we have more control/supervision since we handcraft all python modules.

calbrecht · 2020-05-31T10:24:06Z

Thank you @prusnak for taking care of that. I totally missed that.

12.x is still maintained by Node, but 14.15.0 became the new LTS version of Node.js. As per NixOS#89184, `nodejs` should point to the current LTS. Bump the default `nodejs` version from 12 to 14.

Mic92 and others added 5 commits May 27, 2020 21:45

nodePackages: switch to 12

6602f87

Signed-off-by: Christian Albrecht <christian.albrecht@mayflower.de>

Delete versioned node-packages

ca3f5fe

Change to non-versioned node-packages generate.sh

8a03e80

Change to nodePackages and nodePackages_latest

2312f5f

currently LTS 12.x and Current 14.x

Add meta.broken and meta.platforms to node-packages

db06229

calbrecht requested a review from jonringer as a code owner May 29, 2020 20:17

ofborg bot added the 8.has: documentation label May 29, 2020

calbrecht commented May 29, 2020

View reviewed changes

pkgs/development/node-packages/node-packages.json Outdated Show resolved Hide resolved

calbrecht commented May 29, 2020

View reviewed changes

pkgs/development/node-packages/node-packages.json Outdated Show resolved Hide resolved

Mic92 approved these changes May 29, 2020

View reviewed changes

Readd the two node-packages that were lost in transition and time

6149df8

ofborg bot added 10.rebuild-darwin: 101-500 10.rebuild-linux: 101-500 labels May 29, 2020

Mic92 merged commit 87908c3 into NixOS:master May 30, 2020

calbrecht deleted the continue-pull-85764 branch May 30, 2020 10:45

zowoq mentioned this pull request May 31, 2020

.editorconfig: fix for node-packages #89236

Merged

Mrmaxmeier mentioned this pull request Jun 10, 2020

thelounge: re-apply 'write out default path' #90069

Merged

10 tasks

punkeel mentioned this pull request Oct 28, 2020

nodePackages: Switch to 14.x #101936

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nodePackages: Switch to 12.x and 14.x, continue #85764 #89184

nodePackages: Switch to 12.x and 14.x, continue #85764 #89184

calbrecht commented May 29, 2020

Mic92 left a comment •

edited

calbrecht commented May 30, 2020

prusnak commented May 30, 2020 •

edited

Mic92 commented May 30, 2020 •

edited

calbrecht commented May 30, 2020

Mic92 commented May 30, 2020 •

edited

calbrecht commented May 30, 2020

Mic92 commented May 30, 2020

calbrecht commented May 31, 2020

nodePackages: Switch to 12.x and 14.x, continue #85764 #89184

nodePackages: Switch to 12.x and 14.x, continue #85764 #89184

Conversation

calbrecht commented May 29, 2020

Motivation for this change

Things done

Mic92 left a comment • edited

Choose a reason for hiding this comment

calbrecht commented May 30, 2020

prusnak commented May 30, 2020 • edited

Mic92 commented May 30, 2020 • edited

calbrecht commented May 30, 2020

Mic92 commented May 30, 2020 • edited

calbrecht commented May 30, 2020

Mic92 commented May 30, 2020

calbrecht commented May 31, 2020

Mic92 left a comment •

edited

prusnak commented May 30, 2020 •

edited

Mic92 commented May 30, 2020 •

edited

Mic92 commented May 30, 2020 •

edited