[WIP] Enable Vault UI #49082

arianvp · 2018-10-25T00:38:20Z

Motivation for this change

Since version 0.10, Vault ships with an administrator UI. This PR allows us to use that.
This was a bit of task to get working!

The new consul version also packages its UI this way. So when we want to bump consul from the current (pretty old) version, we can use the same groundwork we did here. So that's good news.

Including the assets has barely any impact on the closure size:

these paths will be fetched (13.92 MiB download, 76.72 MiB unpacked):
  /nix/store/5ii0rjvfj6s9260g1mwyrx8bmyqildh9-vault-0.11.2
copying path '/nix/store/5ii0rjvfj6s9260g1mwyrx8bmyqildh9-vault-0.11.2' from 'https://cache.nixos.org'...
/nix/store/5ii0rjvfj6s9260g1mwyrx8bmyqildh9-vault-0.11.2	   83119048


[arian@ryzen:~/Projects/nixpkgs]$ nix path-info -S $(nix-build -A vault)
/nix/store/dwhii30w2xlbs81mpca2kzx8y3byrmrq-vault-0.11.2	   83779736

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

Enables the vault UI, which ships with vault since version 0.10

Also adds a test that this option actually works

arianvp · 2018-10-25T01:08:55Z

I'm not sure why the build is failing. It gives a "path is not valid" but I can't reproduce that locally

Mic92 · 2018-10-25T10:22:35Z

yarn.nix is a bit large can you try to integrate this into node-packages? Then we can share nix expressions between packages.

arianvp · 2018-10-25T10:50:56Z

The problem is that consul and vault use yarn.lock instead of package-lock.json. Which npm2nix isn't compatible with ... But I could try. Maybe I don't need the exact upstream versions for them.

arianvp · 2018-10-25T11:06:38Z

Another problem is that some of the dependencies of Vault's UI are not on npm but are on github instead. But I could see if I can try to split that up someway.

edit: apparently npm2nix does support this!

I'll have a look today. Lets hope consul and vault use the same package versions :)

arianvp · 2018-10-25T13:27:32Z

Alas @Mic92

I tried adding it to node-packages but I don't think it's a smart idea or feasible. We don't really record version numbers in node-packages at the moment, as we mostly use it to ship "standalone nodejs based applications", not as a set of packages that is used as dependencies for other packages. Both vault and consul need specific versions of npm libraries, not just "any version" so adding it to our global node package set is not a very scalable approach. the node-packages set as currently set up doesn't really seem to be made for this kind of stuff. I think vendoring the needed yarn.nix file with the consul and vault package is the easiest way to go ...

LnL7 · 2018-10-26T00:24:09Z

I think the evaluation error is cause by the way yarn2nix works, it uses import from derivation which isn't allowed on ofborg or hydra.

arianvp · 2018-10-26T09:09:55Z

I disabled import from derivation by explicitly passing a lockfile I thought.

…

On Fri, Oct 26, 2018, 02:24 Daiderd Jordan ***@***.***> wrote: I think the evaluation error is cause by the way yarn2nix works, it uses import from derivation which isn't allowed on ofborg or hydra. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#49082 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAmWo1FoG7q2Sqzn-ujhxx8KuJ3eu4FQks5uolYrgaJpZM4X5Sql> .

LnL7 · 2018-10-26T09:22:08Z

I thought so too, but maybe not? The error definitely looks like an error caused by restrict-eval (see man nix.conf)

arianvp · 2018-10-26T11:48:15Z

Is patterns like:
https://github.com/NixOS/nixpkgs/pull/49082/files#diff-9f45399180b286c35746a0fcafa076cfR25

disallowed in restricted eval mode? I.e. the source of one package is dependent on the source of another package?

arianvp · 2018-10-26T12:04:09Z

Ahaaa... It's importJSON'ing package.json.... I see.

arianvp · 2018-10-26T12:09:27Z

I filed an issue upstream with yarn2nix nix-community/yarn2nix#83

This method isn't allowed on Hydra. It was only used to extract metadata from package.json We remove it, so that the build is pure again. We should probably upstream this to the yarn2nix repo as well

arianvp · 2018-10-26T16:01:08Z

Apparently consul simply vendors the UI assets in their Repo: #48714 (comment) . This makes building consul a breeze.

I've asked upstream if they would consider doing something similar to vault. This means we wouldn't have to vendor 12k lines of yarn dependencies. hashicorp/vault#5622

arianvp · 2018-10-26T17:32:40Z

Unfortunately, they will not change this upstream. So we'll have to go with yarn2nix

arianvp · 2018-10-26T17:46:12Z

@Mic92 are there any technical downsides of including this yarn.nix file that I'm not overseeing, like for example evaluation time or other issues? If not, I would prefer to merge this as is, as putting all these dependencies in node-packages seems non-trivial to me at this moment. For the reasons put above in this thread.

Mic92 · 2018-10-27T07:48:06Z

This pull request adds 500 KB generated data to our repository. This will stick in our history and requires additional storage for each channel user. I can tell from experience that also git does not like large generated text files. Also we might be able to fix nix evaluation performance it would be impossible to fix our history.
To still make progress on this pull request I suggest the following approach:

Fork vault on github
Add travis configuration that builds the ui assets and upload them on release tags.
Change the nix build to just take the compiled asset archive instead.

Here is an example project for uploading release assets: https://github.com/Mic92/cntr/blob/master/.travis.yml#L39

arianvp · 2018-10-27T10:24:15Z

That sounds like a good idea. I will do that. If it turns out to be too much work. I'm just gonna drop this PR, and we'll just continue building Vault without a UI.

arianvp · 2018-12-05T12:04:09Z

I lost interest in getting this merged, as I have no interest in maintaining my own fork of vault with my own CI, given it's quite a security-senstive product. Someone else feel free to pick this up, or use it to build Vault locally with UI enabled :)

This binray contain the UI part of HashiCorp Vault that we were not able to build it due to the need to generate a very big yarn file. NixOS#49082

arianvp added 2 commits October 25, 2018 01:44

vault: Enable vault UI

1b9f03a

Enables the vault UI, which ships with vault since version 0.10

nixos/vault: Add option to enable the vault web ui

174c69e

Also adds a test that this option actually works

arianvp force-pushed the vault-ui branch from c599034 to 174c69e Compare October 25, 2018 00:39

GrahamcOfBorg added 6.topic: nixos 8.has: module (update) labels Oct 25, 2018

vault: Add arianvp to the maintainers list

aac6cd8

arianvp mentioned this pull request Oct 26, 2018

consul: 0.9.3 -> 1.3.0 #48714

Closed

9 tasks

arianvp added 2 commits October 26, 2018 14:49

yarn2nix: Remove lib.importJSON from yarn2nix

64dc388

This method isn't allowed on Hydra. It was only used to extract metadata from package.json We remove it, so that the build is pure again. We should probably upstream this to the yarn2nix repo as well

vault: Use new yarn2nix

03fdfe6

GrahamcOfBorg added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Oct 26, 2018

arianvp mentioned this pull request Oct 26, 2018

vendor vault UI's bindata_assetfs.go like Consul does? hashicorp/vault#5622

Closed

arianvp mentioned this pull request Oct 26, 2018

consul: 0.9.3 -> 1.3.0 with vendored UI #49165

Merged

9 tasks

roberth added the 2.status: work-in-progress label Oct 28, 2018

Mic92 changed the title ~~Enable Vault UI~~ [WIP] Enable Vault UI Oct 28, 2018

Mic92 removed the 2.status: work-in-progress label Oct 28, 2018

Mic92 mentioned this pull request Oct 29, 2018

rebuilding while offline nix-community/home-manager#430

Closed

zimbatm mentioned this pull request Dec 5, 2018

vault: add ui #42614

Closed

arianvp closed this Dec 5, 2018

arianvp mentioned this pull request Jul 7, 2019

nomad: "Nomad UI is not available in this binary." #64410

Closed

PsyanticY mentioned this pull request Jul 20, 2019

vault-bin: init at 1.1.3 #65166

Merged

9 tasks

offlinehacker mentioned this pull request Jul 23, 2019

language package management tooling missing hashes #65275

Open

PsyanticY added a commit to PsyanticY/nixpkgs that referenced this pull request Jul 25, 2019

vault-bin: init at 1.1.3

f403684

This binray contain the UI part of HashiCorp Vault that we were not able to build it due to the need to generate a very big yarn file. NixOS#49082

arianvp mentioned this pull request Apr 15, 2020

Recommend buildGoPackage instead of buildGoModule in the nixpkgs manual #84826

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Enable Vault UI #49082

[WIP] Enable Vault UI #49082

arianvp commented Oct 25, 2018 •

edited

arianvp commented Oct 25, 2018

Mic92 commented Oct 25, 2018

arianvp commented Oct 25, 2018

arianvp commented Oct 25, 2018 •

edited

arianvp commented Oct 25, 2018

LnL7 commented Oct 26, 2018

arianvp commented Oct 26, 2018 via email

LnL7 commented Oct 26, 2018

arianvp commented Oct 26, 2018

arianvp commented Oct 26, 2018

arianvp commented Oct 26, 2018

arianvp commented Oct 26, 2018

arianvp commented Oct 26, 2018 •

edited

arianvp commented Oct 26, 2018 •

edited

Mic92 commented Oct 27, 2018 •

edited

arianvp commented Oct 27, 2018 •

edited

arianvp commented Dec 5, 2018 •

edited

[WIP] Enable Vault UI #49082

[WIP] Enable Vault UI #49082

Conversation

arianvp commented Oct 25, 2018 • edited

Motivation for this change

Things done

arianvp commented Oct 25, 2018

Mic92 commented Oct 25, 2018

arianvp commented Oct 25, 2018

arianvp commented Oct 25, 2018 • edited

arianvp commented Oct 25, 2018

LnL7 commented Oct 26, 2018

arianvp commented Oct 26, 2018 via email

LnL7 commented Oct 26, 2018

arianvp commented Oct 26, 2018

arianvp commented Oct 26, 2018

arianvp commented Oct 26, 2018

arianvp commented Oct 26, 2018

arianvp commented Oct 26, 2018 • edited

arianvp commented Oct 26, 2018 • edited

Mic92 commented Oct 27, 2018 • edited

arianvp commented Oct 27, 2018 • edited

arianvp commented Dec 5, 2018 • edited

arianvp commented Oct 25, 2018 •

edited

arianvp commented Oct 25, 2018 •

edited

arianvp commented Oct 26, 2018 •

edited

arianvp commented Oct 26, 2018 •

edited

Mic92 commented Oct 27, 2018 •

edited

arianvp commented Oct 27, 2018 •

edited

arianvp commented Dec 5, 2018 •

edited