Added fetchRepoProject #24239

spacekitteh · 2017-03-23T04:40:58Z

Created a "fetchGitRepo" to fetch source repositories managed by Google's git-repo

One problem is fetching clone.bundle 's fails due to python's urllib not being able to parse the result given by repo's call to git config --get-regexp url.*.insteadof, but this behaviour doesn't manifest when calling it from a normal shell, even in nixos.

Mic92 · 2017-03-23T05:31:47Z

The name is ambiguous and people could it mixed it up with fetchgit. Maybe fetchGoogleGitRepo or fetchAndroidGitRepo?

Mic92 · 2017-03-23T05:36:47Z

pkgs/build-support/fetchgitrepo/default.nix

+  done
+
+  export HOME=.repo
+  repo init --manifest-url=${manifest} --manifest-branch=${rev} --depth=1 --no-clone-bundle''


these manifests always points to fixed revisions in other repos?

Yep (or at least, they're supposed to)

I think repo supports floating branch refs too, so that's a risk...

probably this could be decided from case to case. But a fair warning would helpful.

How should the warning be presented?

edolstra · 2017-03-23T10:47:58Z

pkgs/build-support/fetchgitrepo/default.nix

+stdenv.mkDerivation {
+  inherit name cacert;
+  buildCommand = with stdenv.lib; ''
+  mkdir ./.repo


Indentation is wrong here.

What should it be?

jb55 · 2017-03-23T13:48:26Z

This was my initial reaction as well. I haven't heard of git-repo until now. +1 for fetchGoogleGitRepo.

spacekitteh · 2017-03-24T02:07:17Z

I have renamed it to fetchRepoProject, I think that's a far better name

copumpkin · 2017-03-24T02:44:14Z

pkgs/build-support/fetchrepoproject/default.nix

+assert repoRepoRev != "" -> repoRepoURL != "";
+
+stdenv.mkDerivation {
+    inherit name cacert;


Indentation seems kinda wonky across this file

copumpkin · 2017-03-24T02:45:25Z

Looks fine to me apart from the indentation being wonky

copumpkin · 2017-03-24T02:48:38Z

Oh I see you asked earlier how it should look. This is how I'd do it, based on conventions I've seen elsewhere in nixpkgs:

{stdenv, git, gitRepo, gnupg ? null, cacert}:

{name, manifest, rev ? "HEAD", sha256 ? "", repoRepoURL ? "", repoRepoRev ? "", referenceDir ? "",
localManifests ? [] }:

assert repoRepoRev != "" -> repoRepoURL != "";

stdenv.mkDerivation {
  inherit name cacert;
  buildCommand = with stdenv.lib; ''
    mkdir ./.repo
    mkdir ./.repo/local_manifests
    for local_manifest in ${concatMapStringsSep " " (x: "${x}") localManifests}
    do
      cp $local_manifest ./.repo/local_manifests/$(stripHash $local_manifest; echo $strippedName)
    done

    export HOME=.repo
    repo init --manifest-url=${manifest} --manifest-branch=${rev} --depth=1 --no-clone-bundle
    ${optionalString (repoRepoURL != "") " --repo-url=${repoRepoURL}"}
    ${optionalString (repoRepoRev != "") " --repo-branch=${repoRepoRev}"}
    ${optionalString (referenceDir != "") " --reference=${referenceDir}"}

    repo sync --jobs=$NIX_BUILD_CORES --current-branch
    rm -rf $out/.repo
  '';

  GIT_SSL_CAINFO = "${cacert}/etc/ssl/certs/ca-bundle.crt";

  impureEnvVars = stdenv.lib.fetchers.proxyImpureEnvVars ++ [
    "GIT_PROXY_COMMAND" "SOCKS_SERVER"
  ];

  buildInputs = [git gitRepo cacert] ++ stdenv.lib.optional (gnupg != null) [gnupg] ;

  outputHashAlgo = "sha256";
  outputHashMode = "recursive";
  outputHash = sha256;

  preferLocalBuild = true;
  enableParallelBuilding = true;
  inherit manifest rev repoRepoURL repoRepoRev referenceDir;
}

Edit: whoops, I screwed it up a bit by attempting to remove the string appends, but it shouldn't be hard to fix.

spacekitteh · 2017-03-24T03:07:02Z

Alright @copumpkin I reformatted it like that (fixing the optionalString bits of course)

copumpkin · 2017-03-24T03:16:21Z

pkgs/build-support/fetchrepoproject/test.nix

+with import <nixpkgs> {};
+let fetcher = import ./default.nix {inherit stdenv cacert
+git gitRepo gnupg;};
+in


Same formatting issues here, sorry 😄

copumpkin · 2017-03-24T03:16:38Z

pkgs/build-support/fetchrepoproject/test.nix

+  name = "foo";
+  manifest = "https://github.com/CopperheadOS/platform_manifest.git";  
+  rev="nougat-mr1.1-release";
+  sha256 = "51rhjsqka5v02zw6k0m4fnwz8xrx70ifar4afsndm88i6fbd3vaw"; #fake hash, don't worry


Why would I worry about the hash? 😄

the hash is just a random hash to test that repo actually does its thing (that manifest branch, without a tag, follows HEAD)... also it fetches about 80GB of sources |-)

I could change it to a smaller project with a fixed rev, though?

Added to grab projects added by git-repo. Contains some problems still reguarding purity and clone.bundle, but good enough for now.

copumpkin · 2017-03-24T03:30:05Z

Looks good to me, thanks!

LnL7 · 2017-03-24T07:45:04Z

This broken evaluation, fixed in f6669da.

spacekitteh · 2017-03-24T08:25:23Z

@LnL7 that seems to have an optionalString on each (bash script) line?

spacekitteh · 2017-03-24T08:27:07Z

Oh shit, that's because I forgot to add the change before committing. What you have at the moment doesn't work, either, @LnL7 .

Mic92 reviewed Mar 23, 2017

View reviewed changes

edolstra reviewed Mar 23, 2017

View reviewed changes

spacekitteh force-pushed the fetchFromGitRepo branch from a3172c3 to e3d3c52 Compare March 24, 2017 02:36

copumpkin reviewed Mar 24, 2017

View reviewed changes

spacekitteh force-pushed the fetchFromGitRepo branch from e3d3c52 to ef50fba Compare March 24, 2017 03:05

spacekitteh force-pushed the fetchFromGitRepo branch from ef50fba to 4afc9e3 Compare March 24, 2017 03:11

copumpkin reviewed Mar 24, 2017

View reviewed changes

spacekitteh force-pushed the fetchFromGitRepo branch 2 times, most recently from 8e6742b to cef00e7 Compare March 24, 2017 03:26

fetchRepoProject: init.

76e095c

Added to grab projects added by git-repo. Contains some problems still reguarding purity and clone.bundle, but good enough for now.

copumpkin changed the title ~~Added fetchGitRepo~~ Added fetchRepoProject Mar 24, 2017

spacekitteh force-pushed the fetchFromGitRepo branch from cef00e7 to 76e095c Compare March 24, 2017 03:29

copumpkin merged commit a9644fb into NixOS:master Mar 24, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added fetchRepoProject #24239

Added fetchRepoProject #24239

spacekitteh commented Mar 23, 2017 •

edited

Mic92 commented Mar 23, 2017 •

edited

Mic92 Mar 23, 2017

spacekitteh Mar 23, 2017

bjornfor Mar 23, 2017

Mic92 Mar 23, 2017 •

edited

spacekitteh Mar 24, 2017

edolstra Mar 23, 2017

spacekitteh Mar 24, 2017

jb55 commented Mar 23, 2017 via email

spacekitteh commented Mar 24, 2017

copumpkin Mar 24, 2017

copumpkin commented Mar 24, 2017

copumpkin commented Mar 24, 2017 •

edited

spacekitteh commented Mar 24, 2017

copumpkin Mar 24, 2017

copumpkin Mar 24, 2017

spacekitteh Mar 24, 2017

copumpkin commented Mar 24, 2017

LnL7 commented Mar 24, 2017

spacekitteh commented Mar 24, 2017

spacekitteh commented Mar 24, 2017

Added fetchRepoProject #24239

Added fetchRepoProject #24239

Conversation

spacekitteh commented Mar 23, 2017 • edited

Mic92 commented Mar 23, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mic92 Mar 23, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jb55 commented Mar 23, 2017 via email

spacekitteh commented Mar 24, 2017

Choose a reason for hiding this comment

copumpkin commented Mar 24, 2017

copumpkin commented Mar 24, 2017 • edited

spacekitteh commented Mar 24, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

copumpkin commented Mar 24, 2017

LnL7 commented Mar 24, 2017

spacekitteh commented Mar 24, 2017

spacekitteh commented Mar 24, 2017

spacekitteh commented Mar 23, 2017 •

edited

Mic92 commented Mar 23, 2017 •

edited

Mic92 Mar 23, 2017 •

edited

copumpkin commented Mar 24, 2017 •

edited