Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mirror: Add packages/options data to channel release. #35

Merged
merged 13 commits into from Mar 26, 2020
Merged

mirror: Add packages/options data to channel release. #35

merged 13 commits into from Mar 26, 2020

Conversation

samueldr
Copy link
Member

@samueldr samueldr commented Mar 24, 2020 •
edited

Though, I heavily suggest you read the commits, there's more to this PR.

  • Adds shell.nix using flake-compat.
  • Allows running in dry-run mode for easier development.
  • Outputs the current running command.
  • Minor cleanups related to system use.

I can reduce the scope of this PR to the basics needed for the json data if desired.

The new version was tested by forcing the download of incomplete evals. Note latest instead of latest-finished!

DRY_RUN=1 ./mirror-nixos-branch.pl nixpkgs-unstable https://hydra.nixos.org/job/nixpkgs/trunk/tarball/latest

The same could not be done for NixOS yet.

This will fail with current channels, as there is no options/packages json file. This was tested using this temp workaround.

diff --git a/mirror-nixos-branch.pl b/mirror-nixos-branch.pl
index 4bdbd65..1c64006 100755
--- a/mirror-nixos-branch.pl
+++ b/mirror-nixos-branch.pl
@@ -207,8 +207,12 @@ if ($bucketReleases && $bucketReleases->head_key("$releasePrefix")) {
     if ($channelName =~ /nixos/) {
         downloadFile("nixos.channel", "nixexprs.tar.xz");
         downloadFile("nixos.iso_minimal.x86_64-linux");
+        eval {
         downloadFile("nixpkgs.tarball", "packages.json.br", "json-br");
+        } or print STDERR "warning: packages.json.br skipped!\n";
+        eval {
         downloadFile("nixos.options", "options.json.br", "json-br");
+        } or print STDERR "warning: options.json.br skipped!\n";
 
         if ($channelName !~ /-small/) {
             downloadFile("nixos.iso_minimal.i686-linux");

I don't like discriminating on buildproducts by using subtype, as I assume subtypes could realistically be duplicated in a build.

Though, we do not have a proper way to choose which build product.

For this build:

This is the buildproducts info:

          'buildproducts' => {
                               '1' => {
                                        'defaultpath' => '',
                                        'sha256hash' => 'd823328e76005677a227c8e5926ee0822cabbd720c8cfa6fecded9ce576a555d',
                                        'type' => 'file',
                                        'filesize' => 2740220,
                                        'name' => 'packages.json.br',
                                        'path' => '/nix/store/97i5pah4vy2x672z1kalh5g125s81gl0-nixpkgs-tarball-20.09pre218220.35b2ad79ff6/packages.json.br',
                                        'subtype' => 'json-br',
                                        'sha1hash' => '7516167caf27a1451ea38489260db1c67f4a07c6'
                                      },
                               '2' => {
                                        'type' => 'file',
                                        'defaultpath' => '',
                                        'sha256hash' => 'a3489657e72d1840a5efe37a65e59053ef83d5b2c31eb6403162f89d0e4c5e94',
                                        'subtype' => 'source-dist',
                                        'path' => '/nix/store/97i5pah4vy2x672z1kalh5g125s81gl0-nixpkgs-tarball-20.09pre218220.35b2ad79ff6/tarballs/nixpkgs-20.09pre218220.35b2ad79ff6.tar.xz',
                                        'name' => 'nixpkgs-20.09pre218220.35b2ad79ff6.tar.xz',
                                        'sha1hash' => '3b8677e95264d2041dafa121a485a007aec92156',
                                        'filesize' => 16762476
                                      }
                             },

The best alternative implementation, and more correct one, would be to allow passing a regex pattern to match on path, so even name could be duplicated. We know that a path can't.

@samueldr samueldr marked this pull request as ready for review March 24, 2020 21:36
@samueldr
Add .gitignore for result
9d6f0ce
@samueldr
Add flake-compat default.nix/shell.nix
55d8093 
Let's keep this repository as accessible to contributors as possible!
@samueldr
Split native programs build from the main build for shell use
f4d0407 
This way, we can fully run the mirror script in dry-run mode.
@samueldr
mirror: Introduce two functions for running commands
ab952e5 
They are thin wrappers over `system`, but two main points:

 * They log the (unescaped) command.
 * `run` will auto-die.

This removes the need to pepper `== 0 or die` in the script.
@samueldr
mirror: Use experimental "nix-command" feature.
f5d04ea
@samueldr
mirror: Add dry-run mode to better test script
84c6a24
@samueldr
mirror: Cosmetic change to the release info output
877e989
@samueldr
mirror: Convert all system use to run*
13154f8
@samueldr
mirror: allow downloadFile to select a specific product
badde7c
@samueldr
mirror: Download packages.json.br and options.json.br
0568ef3
@samueldr
mirror: Use shell-free calls when possible
51104d8 
Passing a list of parameters to `system` automatically gives us peace of
mind with regards to shell escaping. This is because it will not rely on
a shell invocation to run the given parameters, but directly exec to it.

The left-over shell invocations all use redirection.
@garbas garbas requested review from garbas and edolstra March 25, 2020 11:36
garbas
garbas requested changes Mar 25, 2020
View reviewed changes
mirror-nixos-branch.pl Show resolved Hide resolved
@edolstra edolstra merged commit 3cd3da0 into NixOS:master Mar 26, 2020
@samueldr samueldr deleted the feature/channel-json-data branch March 26, 2020 18:19
@samueldr samueldr mentioned this pull request Mar 26, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@garbas garbas garbas requested changes

@edolstra edolstra Awaiting requested review from edolstra

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@samueldr @garbas @nixos-discourse @edolstra