vscode-extensions: Script to generate Nix for "latest" version of all installed vscode extensions #44292

mankyKitty · 2018-08-01T06:09:47Z

Motivation for this change

This is a useful bash script for generating the nix expressions to be used with this expression: https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/editors/vscode/with-extensions.nix#L18

These extensions are fiddly to update by hand as each package must be inspected individually and then the SHA must also be generated before the list can be updated and then the packages installed.

This script only cares about vscode extensions that you already have installed, any new extensions must be inspected and constructed yourself. If you use the bbenoist.Nix extension then you will have to filter that out as this script doesn't force any particular arrangement. The expression to filter the Nix vscode extension is __filter (e: e.name != "Nix").

This is my first PR to nixpkgs so I'm keen to hear if there are better ways of managing this! 👍

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

Mic92 · 2018-08-01T06:33:55Z

pkgs/misc/vscode-extensions/update_installed_exts.sh

+fi
+
+function get_ver() {
+  curl --silent "https://raw.githubusercontent.com/$1/$2/master/package.json" | grep '"version":' | tr -d '[:alpha:]", :'


jq might be more robust here, since version can be split over two lines.

Mic92 · 2018-08-01T06:36:57Z

pkgs/misc/vscode-extensions/update_installed_exts.sh

+    VER=$(get_ver $OWNER $EXT2)
+  fi
+  if [ -z "$VER" ]; then
+    VER="unknown"


Should the script not rather fail in the case the version could not be retrieved?

Mic92 · 2018-08-01T06:37:35Z

pkgs/misc/vscode-extensions/update_installed_exts.sh

@@ -0,0 +1,38 @@
+#! /usr/bin/env bash
+
+CODE=$1


set -eu -o pipefail

is usually a good idea.

Mic92 · 2018-08-01T06:38:46Z

pkgs/misc/vscode-extensions/update_installed_exts.sh

@@ -0,0 +1,38 @@
+#! /usr/bin/env bash


Using nix-shell shebang can ensure that code is always in PATH: http://iam.travishartwell.net/2015/06/17/nix-shell-shebang/

mankyKitty · 2018-08-01T07:05:14Z

Picture me nodding appreciatively. 👍 Those fixes make sense, I'll make it a bit more robust.

coretemp · 2018-08-01T16:54:27Z

I don't see the need to write it in bash over sh.

mankyKitty · 2018-08-02T01:05:28Z

I've tried to account for all the suggestions and just make the script more portable and robust. Thanks for the tip about the nix-shell shebang, that's awesome!

My sh-fu isn't the best, but I think these changes have made it much more reliable. 👍

coretemp · 2018-08-02T08:40:37Z

pkgs/misc/vscode-extensions/update_installed_exts.sh

@@ -0,0 +1,65 @@
+#! /usr/bin/env nix-shell
+#! nix-shell -i sh -p curl jq coreutils


You are specifying that you use sh, but you use keywords like function, which are not in sh.

Which is always bash in nixpkgs, but being explicit about bash is better though.

matthewbauer · 2018-08-03T04:58:54Z

You should be able to set updateScript to this file.

mankyKitty · 2018-08-03T05:59:33Z

@matthewbauer I'm happy to do that, but I'm not familiar with the updateScript property so just to check..

In here somewhere: https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/editors/vscode/with-extensions.nix#L69

I would add something like:

updateScript = ../../../misc/vscode-extensions/update_installed_exts.sh;

Or would I need to link the script in during the derivation install phase so it can just be:

updateScript = ./update_installed_exts.sh;

Apologies, still learning the ropes. :)

coretemp · 2018-08-03T06:24:49Z

pkgs/misc/vscode-extensions/update_installed_exts.sh

+}
+
+function get_ver() {
+    fetch "https://raw.githubusercontent.com/$1/$2/master/package.json" | jq '.version' | tr -d '"'


jq '.version' | tr -d '"' -> jq -r .version

Neat, thanks

Mic92 · 2018-08-03T07:44:44Z

pkgs/misc/vscode-extensions/update_installed_exts.sh

+if [ $# -ne 0 ]; then
+    CODE=$1
+else
+    CODE=$(which code)


which might be not in PATH by default, command -v code should work.

Mic92 · 2018-08-03T07:45:12Z

pkgs/misc/vscode-extensions/update_installed_exts.sh

+}
+
+function get_ver() {
+    fetch "https://raw.githubusercontent.com/$1/$2/master/package.json" | jq '.version' | tr -d '"'


jq -r '.version' makes tr unnecessary

Mic92 · 2018-08-03T07:48:32Z

pkgs/misc/vscode-extensions/update_installed_exts.sh

+}
+
+function fetch() {
+    RES=$(curl --silent "$1")


--silent --show-error --fail and fail on error is better then checking for 404 because it also catch other server errors (500) and network failure.

I've removed that function and reworked get_ver to be more adept at handling errors.

worldofpeace · 2018-08-04T02:59:21Z

pkgs/misc/vscode-extensions/update_installed_exts.sh

 }

+# Try to fetch the SHA256 of the extension bundle via the VisualStudio Marketplace "API".


Is it possible to do something similar to get the version?

I spent some time trying to do exactly that and I haven't been able to manage it. I couldn't find an API for interrogating the package information either. The thing I haven't pursued is pulling apart the package once it's been downloaded. Mainly because that felt like tugging on the thread of creating proper derivations for extensions, which I haven't felt up to just yet.

Sorry this is sort of turning into a bit of an epic PR for something so unremarkable, I do appreciate all the help and patience. 👍

We get some really good code reviews in here at nixpkgs. 😄

In the event that the extensions version can't be found it would be nice if it didn't fail and continued on to the next extension. Maybe just leave that particular expression incomplete (missing the version) and then print at the end what extension/s a version couldn't be found?

I'm not sure about that, this is meant as just a helpful tool so I can see that just stubbing in a "unknown_version" in that field and reporting might be nice. But it doesn't feel like the 'nix way' to me.

This might be used in an automated fashion, I'd rather it fail and let the human know than either introduce unexpected behaviour with defaulted string values, or even worse silently uninstalling packages (unlikely, but I worry).

Although I could make that behaviour optional... I'll have a go.

Yeah absolutely. Maybe call it --keep-going.

I can more reliably get the version with a single request now, I'll tidy this up a bit more!

worldofpeace · 2018-08-05T03:29:38Z

pkgs/misc/vscode-extensions/update_installed_exts.sh

+    # Unpack the file we need (no need to unpack everything)
+    unzip -qd "$EXTTMP/$N" "$EXTTMP/$N.zip" "extension/package.json"
+    # Pull out the version
+    VER=$(jq -r '.version' < "$EXTTMP/$N/extension/package.json")


worldofpeace · 2018-08-05T07:31:41Z

pkgs/misc/vscode-extensions/update_installed_exts.sh

+    # Pull out the version
+    VER=$(jq -r '.version' < "$EXTTMP/$N/extension/package.json")
+    # Calculate the sha and trim the zip path from the output.
+    SHA=$(sha256sum < "$EXTTMP/$N.zip" | awk '{ print $1 }')


You can just use nix-hash --flat --base32 --type sha256 for the hashes.

worldofpeace · 2018-08-06T04:09:33Z

pkgs/misc/vscode-extensions/update_installed_exts.sh

+    curl --silent --show-error --fail -X GET -o "$EXTTMP/$N.zip" "$URL"
+    # Unpack the file we need to stdout then pull out the version
+    VER=$(jq -r '.version' <(unzip -qc "$EXTTMP/$N.zip" "extension/package.json"))
+    # Calculate the sha and trim the zip path from the output.


"Calculate the sha and trim the zip path from the output." -> "Calculate the sha"

worldofpeace

This all seems nice to me, thanks for writing this 👍

Can you use git rebase -i and squash the commits into the first one and change the commit message to something like "vscode-extensions: blah"?

Mic92 · 2018-08-06T10:43:12Z

pkgs/misc/vscode-extensions/update_installed_exts.sh

+    N="$1.$2"
+
+    # Create a tempdir for the extension download
+    EXTTMP=$(mktemp -d -t vscode_exts_XXXXXXXX)


If the script is aborted this, it would leave a temporary directory.
You can install an error handler to cleanup the directory when the function returns

EXTTMP=$(mktemp -d -t vscode_exts_XXXXXXXX) trap "rm -rf $EXTTMP; trap - RETURN EXIT" RETURN EXIT

Just saw clean_up. Both ways work. Also the local trap is more precise.

mankyKitty · 2018-08-06T23:09:10Z

Rebase applied, thanks for the help everyone. 👍

infinisil · 2018-08-06T23:35:59Z

pkgs/misc/vscode-extensions/update_installed_exts.sh

+for i in $($CODE --list-extensions)
+do
+    OWNER=$(echo "$i" | awk -F '.' '{ print $1 }';)
+    EXT=$(echo "$i" | awk -F '.' '{ print $2 }';)


You can just use cut -d. -f1 and -f2 for this. Then awk also won't be a dependency anymore (it's currently not declared in the nix-shellbang even).

infinisil · 2018-08-06T23:46:17Z

pkgs/misc/vscode-extensions/update_installed_exts.sh

+    rm -Rf "$EXTTMP"
+    # I don't like 'rm -Rf' lurking in my scripts but this seems appropriate
+
+    printf "  {\n    name = \"%s\";\n    publisher = \"%s\";\n    version = \"%s\";\n    sha256 = \"%s\";\n  }\n" "$2" "$1" "$VER" "$SHA"


I think this will work as well and looks better:

cat <<-EOF { name = "$2"; publisher = "$1"; version = "$VER"; sha256 = "$SHA"; } EOF

Thanks for that, i was trying to bang something like that into shape but couldn't make it stick. Ta. :)

infinisil · 2018-08-07T00:12:23Z

~~Rebase~~Squash into 1 commit and this is good to merge :)

… installed vscode extensions Added better practices to update_exts script. Use `jq` instead of `grep` for more reliable JSON querying. Check for 404 when requesting package.json information to avoid mangled output. Added proper failure points for missing vscode package, unknown version, and if the code executable couldn't be found. Switched to using a `nix-shell` shebang for even better reliability and use the `sh` shell to be that little bit more generic. Script is still clunky and sequential, anything more and I'd need to write a proper program to do this and that's getting a bit silly? But people that have a dozen or so extensions might be in for a long wait. Be explicit about using bash Improve the use of jq to remove unnecessary use of tr. Hat-tip coretemp. Add some comments, finally. Remove the `fetch` function. Change the `get_ver` function to more accurately demonstrate what it is trying to do, as well as add in some better error handling for non-200 http responses. I couldn't make the bash `${param/search/replacement}` work for chopping up the response in the `get_ver` function, hence the use of `sed`. Hopefully it all makes a bit more sense now. Remove github requests. VSIXPackage is just a zip format in disguise so use a tmpdir and unpackage the package.json file for the file in question so we can get the precise version that we're interested in without additional redundant calls to github that may not provide the right answer anyway. Add trap to try to clean up the temp folders and clean up as we go. I can't use 'fetchurl' or even 'nix-prefetch-url' because for the former we don't yet know the hash that we're after and for the latter there isn't a way to tie the predownloaded file into the next part of the workflow. Prevent an unnecessary file from being extracted. Change the unzip command to read the file we're after to stdout so we can use jq on it directly instead of creating a file, reading it, then deleting it. Courtesy of worldofpeace, remove the dependency on coreutils and use the provided nix-hash function to generate the required hash. Fix up a comment Remove use of 'awk' and clean up individual Nix printing with cat to EOF expression.

mankyKitty · 2018-08-07T00:19:42Z

Commits have been squished! Phew, this turned into a bit of an epic. 👍

Depending on how these are merged, the squash be done at merge time on the github side. There is an option for "Squash and Merge", can save some hassle when single commit merges are helpful.

infinisil · 2018-08-07T01:22:48Z

True.. Maybe I should use that more. On the other hand I feel like it's the contributors job to do the squashing, it's like a "finishing up everything", along with potentially combining commit messages and such.

Thanks for the PR!

GrahamcOfBorg added 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Aug 1, 2018

Mic92 reviewed Aug 1, 2018

View reviewed changes

puffnfresh approved these changes Aug 2, 2018

View reviewed changes

coretemp reviewed Aug 2, 2018

View reviewed changes

worldofpeace approved these changes Aug 2, 2018

View reviewed changes

coretemp reviewed Aug 3, 2018

View reviewed changes

Mic92 reviewed Aug 3, 2018

View reviewed changes

worldofpeace reviewed Aug 4, 2018

View reviewed changes

worldofpeace reviewed Aug 5, 2018

View reviewed changes

worldofpeace reviewed Aug 6, 2018

View reviewed changes

worldofpeace approved these changes Aug 6, 2018

View reviewed changes

Mic92 reviewed Aug 6, 2018

View reviewed changes

mankyKitty changed the title ~~Added bash to generate nix for updating installed vscode-extensions~~ vscode-extensions: Script to generate Nix for "latest" version of all installed vscode extensions Aug 6, 2018

mankyKitty force-pushed the vscode-exts-update-script branch from dce9a10 to ea7dc55 Compare August 6, 2018 23:08

infinisil requested changes Aug 6, 2018

View reviewed changes

infinisil reviewed Aug 6, 2018

View reviewed changes

infinisil approved these changes Aug 7, 2018

View reviewed changes

mankyKitty force-pushed the vscode-exts-update-script branch from a23601e to 9321785 Compare August 7, 2018 00:15

infinisil merged commit 308d780 into NixOS:master Aug 7, 2018

Janik-Haag added the 12. first-time contribution label Jun 12, 2023

		@@ -0,0 +1,65 @@
		#! /usr/bin/env nix-shell
		#! nix-shell -i sh -p curl jq coreutils

		}

		# Try to fetch the SHA256 of the extension bundle via the VisualStudio Marketplace "API".

vscode-extensions: Script to generate Nix for "latest" version of all installed vscode extensions #44292

vscode-extensions: Script to generate Nix for "latest" version of all installed vscode extensions #44292

Conversation

mankyKitty commented Aug 1, 2018

Motivation for this change

Things done

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mankyKitty commented Aug 1, 2018

coretemp commented Aug 1, 2018

mankyKitty commented Aug 2, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthewbauer commented Aug 3, 2018

mankyKitty commented Aug 3, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

worldofpeace Aug 4, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

worldofpeace left a comment

Choose a reason for hiding this comment

Mic92 Aug 6, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mankyKitty commented Aug 6, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

infinisil commented Aug 7, 2018 • edited

mankyKitty commented Aug 7, 2018

infinisil commented Aug 7, 2018

worldofpeace Aug 4, 2018 •

edited

Mic92 Aug 6, 2018 •

edited

infinisil commented Aug 7, 2018 •

edited