git: 2.16.2 -> 2.17.0 #38636

layus · 2018-04-09T08:07:02Z

Also refreshed and fixed the patch queue.

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

layus · 2018-04-09T11:20:40Z

Pinging @peti, @the-kenny and @wmertens as maintainers. I did my best to port the patches, but I cannot possibly (locally) compile the 500+ packages that depend on git.

peti · 2018-04-09T11:44:56Z

@GrahamcOfBorg build gitAndTools.git-annex

GrahamcOfBorg · 2018-04-09T11:45:37Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: gitAndTools.git-annex

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-04-09T12:00:32Z

Success on x86_64-linux (full log)

Attempted: gitAndTools.git-annex

Partial log (click to expand)

gzipping man pages under /nix/store/9dc2m3nsfj9saks7ysgkvbz02dh91xp8-git-annex-6.20180316/share/man/
strip is /nix/store/fzcs0fn6bb04m82frhlb78nc03ny3w55-binutils-2.28.1/bin/strip
stripping (with command strip and flags -S) in /nix/store/9dc2m3nsfj9saks7ysgkvbz02dh91xp8-git-annex-6.20180316/bin
patching script interpreter paths in /nix/store/9dc2m3nsfj9saks7ysgkvbz02dh91xp8-git-annex-6.20180316
checking for references to /build in /nix/store/9dc2m3nsfj9saks7ysgkvbz02dh91xp8-git-annex-6.20180316...
shrinking RPATHs of ELF executables and libraries in /nix/store/iwdmjklfsv5bl2f382g46dzbb0r84mma-git-annex-6.20180316-doc
strip is /nix/store/fzcs0fn6bb04m82frhlb78nc03ny3w55-binutils-2.28.1/bin/strip
patching script interpreter paths in /nix/store/iwdmjklfsv5bl2f382g46dzbb0r84mma-git-annex-6.20180316-doc
checking for references to /build in /nix/store/iwdmjklfsv5bl2f382g46dzbb0r84mma-git-annex-6.20180316-doc...
/nix/store/9dc2m3nsfj9saks7ysgkvbz02dh91xp8-git-annex-6.20180316

GrahamcOfBorg · 2018-04-09T12:04:11Z

Success on x86_64-darwin (full log)

Attempted: gitAndTools.git-annex

Partial log (click to expand)

post-installation fixup
Moving /nix/store/nimj3gy33a9yw048psdwinb59hmilfzd-git-annex-6.20180316/share/doc to /nix/store/1rafg2b43kk4052qrip7invl4ay6fzzw-git-annex-6.20180316-doc/share/doc
rmdir: failed to remove '/nix/store/nimj3gy33a9yw048psdwinb59hmilfzd-git-annex-6.20180316/share': Directory not empty
gzipping man pages under /nix/store/nimj3gy33a9yw048psdwinb59hmilfzd-git-annex-6.20180316/share/man/
strip is /nix/store/0fzpxnsanc02i4jsb1yhchjp4p62b2n3-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/nimj3gy33a9yw048psdwinb59hmilfzd-git-annex-6.20180316/lib  /nix/store/nimj3gy33a9yw048psdwinb59hmilfzd-git-annex-6.20180316/bin
patching script interpreter paths in /nix/store/nimj3gy33a9yw048psdwinb59hmilfzd-git-annex-6.20180316
strip is /nix/store/0fzpxnsanc02i4jsb1yhchjp4p62b2n3-cctools-binutils-darwin/bin/strip
patching script interpreter paths in /nix/store/1rafg2b43kk4052qrip7invl4ay6fzzw-git-annex-6.20180316-doc
/nix/store/nimj3gy33a9yw048psdwinb59hmilfzd-git-annex-6.20180316

LnL7 · 2018-04-10T08:44:52Z

I think this broke a bunch of stuff.

Can't locate Git.pm in @INC (you may need to install the Git module) (@INC contains: /nix/store/2jnfwr4gmqp49bdpxkwbva940f6m8bff-git-2.17.0/lib/perl5/site_perl /nix/store/1c0dmikm575as2jqfi07zydwzfpf5jzy-perl-libwww-perl-6.15/lib/perl5/site_perl/5.24.3/x86_64-linux-thread-multi /nix/store/1c0dmikm575as2jqfi07zydwzfpf5jzy-perl-libwww-perl-6.15/lib/perl5/site_perl/5.24.3 /nix/store/1c0dmikm575as2jqfi07zydwzfpf5jzy-perl-libwww-perl-6.15/lib/perl5/site_perl /nix/store/64zx746z8006ihsvznvkdvjpkbhfcsjq-perl-URI-1.73/lib/perl5/site_perl/5.24.3/x86_64-linux-thread-multi /nix/store/64zx746z8006ihsvznvkdvjpkbhfcsjq-perl-URI-1.73/lib/perl5/site_perl/5.24.3 /nix/store/64zx746z8006ihsvznvkdvjpkbhfcsjq-perl-URI-1.73/lib/perl5/site_perl /nix/store/20xahj4zwagnm2bgxl1myxs7v2d7khf6-perl-TermReadKey-2.37/lib/perl5/site_perl/5.24.3/x86_64-linux-thread-multi /nix/store/20xahj4zwagnm2bgxl1myxs7v2d7khf6-perl-TermReadKey-2.37/lib/perl5/site_perl/5.24.3 /nix/store/20xahj4zwagnm2bgxl1myxs7v2d7khf6-perl-TermReadKey-2.37/lib/perl5/site_perl /nix/store/ggb7k5x9855j10dz99467djx4rplg32b-perl-5.24.3/lib/perl5/site_perl/5.24.3/x86_64-linux-thread-multi /nix/store/ggb7k5x9855j10dz99467djx4rplg32b-perl-5.24.3/lib/perl5/site_perl/5.24.3 /nix/store/ggb7k5x9855j10dz99467djx4rplg32b-perl-5.24.3/lib/perl5/5.24.3/x86_64-linux-thread-multi /nix/store/ggb7k5x9855j10dz99467djx4rplg32b-perl-5.24.3/lib/perl5/5.24.3 .) at /nix/store/2jnfwr4gmqp49bdpxkwbva940f6m8bff-git-2.17.0/libexec/git-core/.git-add--interactive-wrapped line 7.
BEGIN failed--compilation aborted at /nix/store/2jnfwr4gmqp49bdpxkwbva940f6m8bff-git-2.17.0/libexec/git-core/.git-add--interactive-wrapped line 7.

peti · 2018-04-10T10:00:38Z

@LnL7, it would be helpful to know how to reproduce that error.

LnL7 · 2018-04-10T17:27:20Z

Sorry, was busy with other stuff. I ran into this with git-add, but I suspect more of the perl commands might be broken.

git init /tmp/foo

# Before
nix run -f https://github.com/NixOS/nixpkgs/archive/c78c7645478ba95f8c654ed79e63a6e7f715c82d.tar.gz git -c git -C /tmp/foo add -p
No changes.

# After
nix run -f https://github.com/NixOS/nixpkgs/archive/1cdcc19218a45f460d9e13bc930f271ed81b5fa7.tar.gz git -c git -C /tmp/foo add -p
Can't locate Git.pm in @INC (you may need to install the Git module) (@INC contains: /nix/store/nkcj1l3cbx3p0ahiy2cc7qbl5d8hqwky-git-2.17.0/lib/perl5/site_perl /nix/store/cfffjc7p8fs9psfbkjc4pcajz15vd367-perl-libwww-perl-6.15/lib/perl5/site_perl/5.24.3/darwin-2level /nix/store/cfffjc7p8fs9psfbkjc4pcajz15vd367-perl-libwww-perl-6.15/lib/perl5/site_perl/5.24.3 /nix/store/cfffjc7p8fs9psfbkjc4pcajz15vd367-perl-libwww-perl-6.15/lib/perl5/site_perl /nix/store/amxjz63msjakpy8x77flsv2d6gn5x67k-perl-URI-1.73/lib/perl5/site_perl/5.24.3/darwin-2level /nix/store/amxjz63msjakpy8x77flsv2d6gn5x67k-perl-URI-1.73/lib/perl5/site_perl/5.24.3 /nix/store/amxjz63msjakpy8x77flsv2d6gn5x67k-perl-URI-1.73/lib/perl5/site_perl /nix/store/bvhv30kdbr1kv82jmi0dq3ri7dxy3dp8-perl-TermReadKey-2.37/lib/perl5/site_perl/5.24.3/darwin-2level /nix/store/bvhv30kdbr1kv82jmi0dq3ri7dxy3dp8-perl-TermReadKey-2.37/lib/perl5/site_perl/5.24.3 /nix/store/bvhv30kdbr1kv82jmi0dq3ri7dxy3dp8-perl-TermReadKey-2.37/lib/perl5/site_perl /nix/store/5spv9lcjwyliipv0yqiv5g8vz0g93v8s-perl-5.24.3/lib/perl5/site_perl/5.24.3/darwin-2level /nix/store/5spv9lcjwyliipv0yqiv5g8vz0g93v8s-perl-5.24.3/lib/perl5/site_perl/5.24.3 /nix/store/5spv9lcjwyliipv0yqiv5g8vz0g93v8s-perl-5.24.3/lib/perl5/5.24.3/darwin-2level /nix/store/5spv9lcjwyliipv0yqiv5g8vz0g93v8s-perl-5.24.3/lib/perl5/5.24.3 .) at /nix/store/nkcj1l3cbx3p0ahiy2cc7qbl5d8hqwky-git-2.17.0/libexec/git-core/.git-add--interactive-wrapped line 7.
BEGIN failed--compilation aborted at /nix/store/nkcj1l3cbx3p0ahiy2cc7qbl5d8hqwky-git-2.17.0/libexec/git-core/.git-add--interactive-wrapped line 7.

peti · 2018-04-10T17:49:25Z

@layus, is there a quick and easy way to fix this issue? If not, we'll have to revert I suppose.

LnL7 · 2018-04-10T19:07:51Z

Looked into this a little bit, the location of the perl libs has moved so the wrappers are not pointing to the right location.

/nix/store/nkcj1l3cbx3p0ahiy2cc7qbl5d8hqwky-git-2.17.0/share/perl5/Git.pm
/nix/store/d8v4fffbczasazfvr561dwcyk5xbyz3p-git-2.16.2/lib/perl5/site_perl/5.24.3/Git.pm

peti · 2018-04-11T07:53:08Z

Reverted in e4fd054.

This reverts commit 5d65b4e, because it broke git-add. See #38636 (comment) for details.

layus · 2018-04-11T11:07:35Z

Ok, Thanks @peti for reverting in the meantime. It worked on NixOS because of PERL5LIB var that was set, referencing the installed git.

To reproduce, I had to call PREL5LIB= nix run -f https://github.com/NixOS/nixpkgs/archive/1cdcc19218a45f460d9e13bc930f271ed81b5fa7.tar.gz git -c git -C /tmp/foo add -p. There should really be a --pure flag to nix run.

The issue is that git moved to a handwritten perl install, and moved the default install path in the meantime. I decided to install perl files to the old location (lib/perl5/site_perl) because this will avoid breaking packages that depend on Git.pm in that location.

@LnL7 I was able to reproduce your bug and fix it, but would you mind testing this again ?

dtzWill · 2018-04-11T17:49:54Z

Guys!! GUYS! All of this could (should) have been avoided, silly rabbits. This update was reverted twice before this PR for this exact reason... (and now three times)... #38387 etc.

We can do better! :).

Thanks for fixing it properly, though (#38763).

layus · 2018-04-11T20:55:56Z

@dtzWill Now, how would you have avoided it ? I was indeed quite ashamed by this abysmal performance but could not find a perfect way to avoid it in the future.

GitHub search really failed me here. "git 2.17" did not give anything meaningful. It turns out that "git 2.17.0" does. Even better would be to search for "git 2.17.0 in:title", but it gives about the same results. So this could be seen as my mistake, at least partially.

We could write a bot to detect identical PRs. It should be quite simple, and entertaining.

Another way to look at it could be to have better tests. I think people are afraid of them because they can discard a whole valid package compilation. This means we need more tests, and we need to make them independent from packages compilation. See the excellent talk from @Profpatsch at NixCon: "Test ALL the things" (youtube) (slides)

/cc @Profpatsch because this could be seen as a good use-case or example.

dtzWill · 2018-04-11T23:39:47Z

@dtzWill Now, how would you have avoided it ? I was indeed quite ashamed by this abysmal performance but could not find a perfect way to avoid it in the future.

I'm not sure, sorry for any implications it was a failure on your part (specifically or otherwise).
I was just frustrated, hopefully understandably so, that my mistake (missing the perl problems) failed to prevent others from wasting time on it, repeatedly. It seems like something that could be avoided, although as you point out it's difficult to identify any particular point of failure.

That said, this round got the issue more attention and some excellent investigation, so maybe it was for the best? 😁

Sounds like a bot would be best for detecting duplicate PR's, I certainly don't think you did anything wrong given your search query. Would this be a reasonable feature for @GrahamcOfBorg to grow?

I agree tests would help here (catch the broken behavior). I believe I've seen that talk, but will look again to make sure and revisit its points :).

wmertens · 2018-04-12T04:33:02Z

Yes, the previous two PRs could have been transformed into tests preventing the broken behavior.

…

On Thu, Apr 12, 2018, 1:39 AM Will Dietz, ***@***.***> wrote: @dtzWill <https://github.com/dtzWill> Now, how would you have avoided it ? I was indeed quite ashamed by this abysmal performance but could not find a perfect way to avoid it in the future. I'm not sure, sorry for any implications it was a failure on your part (specifically or otherwise). I was just frustrated, hopefully understandably so, that my mistake (missing the perl problems) failed to prevent others from wasting time on it, repeatedly. It seems like something that could be avoided, although as you point out it's difficult to identify any particular point of failure. That said, this round got the issue more attention and some excellent investigation, so maybe it was for the best? 😁 ------------------------------ Sounds like a bot would be best for detecting duplicate PR's, I certainly don't think you did anything wrong given your search query. Would this be a reasonable feature for @GrahamcOfBorg <https://github.com/GrahamcOfBorg> to grow? ------------------------------ I agree tests would help here (catch the broken behavior). I believe I've seen that talk, but will look again to make sure and revisit its points :). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#38636 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADWlgkIxt5njm897oHBIkPVkOuNWWflks5tnpRGgaJpZM4TMHIk> .

layus · 2018-04-12T07:55:25Z

That said, this round got the issue more attention and some excellent investigation, so maybe it was for the best? 😁

Yes, I had the same idea but did not dare write it here 😉.

As a bonus, here is an ASCII art of our epic mess. See how the staging/master sync merges reverted "git: 2.16.2 -> 2.16.3" ?

hash | git version | commit message

* 281775bb1a0 2.16.3 git: 2.16.2 -> 2.16.3
| * 2ddba49f59c 2.17.0 git: fix perl libs path
| * ad076ff7ce5 2.17.0 git: 2.16.3 -> 2.17.0
| * 260194193c3 2.16.3 git: 2.16.2 -> 2.16.3
|/  
*   ee6894ca129 2.16.2 Merge staging into master
|\  
| *   0aa59a08d65 2.17.0 Merge master into staging
| |\  
| * | 2bb28e1f019 2.16.3 Revert "git: 2.16.3 -> 2.17.0"
| * | 3bd496faa88 2.17.0 git: 2.16.3 -> 2.17.0
| * | 2b29239eff3 2.16.3 Revert "git: 2.16.3 -> 2.17.0"
| * | dee77f7e433 2.17.0 git: 2.16.3 -> 2.17.0
| * | dcaea58eb72 2.16.3 git: 2.16.2 -> 2.16.3
* | | e4fd05449ed 2.16.2 git: revert "2.16.2 -> 2.17.0"
| |/  
|/|   
* | 5d65b4ebeb6 2.17.0 git: 2.16.2 -> 2.17.0
|/  
* b34274d3638 2.16.2 git: 2.16.1 -> 2.16.2

This is a security update, see [1]. It is not backported from master because master is at 2.17.x after NixOS#38636. [1] https://github.com/git/git/blob/master/Documentation/RelNotes/2.17.1.txt

periklis · 2018-05-31T08:25:26Z

It seems that git 2.17 is still broken on darwin:

Test Summary Report
-------------------
t0050-filesystem.sh                              (Wstat: 256 Tests: 10 Failed: 2)
  Failed tests:  9-10
  Non-zero exit status: 1
Files=802, Tests=18584, 2284 wallclock secs ( 5.35 usr  2.12 sys + 848.76 cusr 1032.68 csys = 1888.91 CPU)
Result: FAIL

dtzWill · 2018-05-31T10:55:42Z

Don't suppose 2.17.1 is any better? If not, we may want to disable the tests on Darwin, or at least that one. Wonder why it fails? :)

periklis · 2018-05-31T11:10:10Z

2.17.1 is also failing on Darwin. However, git-scm provides only 2.17.0 for macOS as a download binary On 31. May 2018, at 12:55, Will Dietz <notifications@github.com> wrote: Don't suppose 2.17.1 is any better? If not, we may want to disable the tests on Darwin, or at least that one. Wonder why it fails? :) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#38636 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAJS-AcCi9GJV2rwRv_i8G2V8KMBJuyIks5t38wvgaJpZM4TMHIk> .

kirelagin · 2018-06-29T16:27:23Z

I’ve got actually more failures with 2.17.1:

Test Summary Report
-------------------
t0050-filesystem.sh                              (Wstat: 256 Tests: 10 Failed: 2)
  Failed tests:  9-10
  Non-zero exit status: 1
t3200-branch.sh                                  (Wstat: 256 Tests: 137 Failed: 1)
  Failed test:  137
  Non-zero exit status: 1
t5601-clone.sh                                   (Wstat: 256 Tests: 102 Failed: 1)
  Failed test:  98
  Non-zero exit status: 1
t6039-merge-ignorecase.sh                        (Wstat: 256 Tests: 2 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
Files=802, Tests=18590, 1999 wallclock secs ( 9.11 usr  3.80 sys + 1602.99 cusr 2950.18 csys = 4566.08 CPU)
Result: FAIL

nixos-discourse · 2020-02-06T22:20:50Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/add-bin-bash-to-avoid-unnecessary-pain/5673/38

git: 2.16.2 -> 2.17.0

5d65b4e

GrahamcOfBorg added 8.has: documentation 10.rebuild-darwin: 501+ 10.rebuild-linux: 501+ labels Apr 9, 2018

peti merged commit 1cdcc19 into NixOS:master Apr 9, 2018

peti added a commit that referenced this pull request Apr 11, 2018

git: revert "2.16.2 -> 2.17.0"

e4fd054

This reverts commit 5d65b4e, because it broke git-add. See #38636 (comment) for details.

layus mentioned this pull request Apr 11, 2018

git: 2.16.2 -> 2.17.0 #38763

Merged

8 tasks

LnL7 mentioned this pull request Apr 11, 2018

nixos: remove PKG_CONFIG_PATH/PERL5LIB environment variables #38791

Merged

8 tasks

FRidh mentioned this pull request May 29, 2018

git: 2.17.0 -> 2.17.1 #41223

Closed

8 tasks

orivej mentioned this pull request May 30, 2018

git: 2.16.2 -> 2.16.4 (for release-18.03) #41244

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

git: 2.16.2 -> 2.17.0 #38636

git: 2.16.2 -> 2.17.0 #38636

layus commented Apr 9, 2018

layus commented Apr 9, 2018

peti commented Apr 9, 2018

GrahamcOfBorg commented Apr 9, 2018

GrahamcOfBorg commented Apr 9, 2018

GrahamcOfBorg commented Apr 9, 2018

LnL7 commented Apr 10, 2018

peti commented Apr 10, 2018

LnL7 commented Apr 10, 2018

peti commented Apr 10, 2018

LnL7 commented Apr 10, 2018

peti commented Apr 11, 2018

layus commented Apr 11, 2018

dtzWill commented Apr 11, 2018

layus commented Apr 11, 2018

dtzWill commented Apr 11, 2018

wmertens commented Apr 12, 2018 via email

layus commented Apr 12, 2018

periklis commented May 31, 2018

dtzWill commented May 31, 2018

periklis commented May 31, 2018 via email

kirelagin commented Jun 29, 2018

nixos-discourse commented Feb 6, 2020

git: 2.16.2 -> 2.17.0 #38636

git: 2.16.2 -> 2.17.0 #38636

Conversation

layus commented Apr 9, 2018

layus commented Apr 9, 2018

peti commented Apr 9, 2018

GrahamcOfBorg commented Apr 9, 2018

GrahamcOfBorg commented Apr 9, 2018

GrahamcOfBorg commented Apr 9, 2018

LnL7 commented Apr 10, 2018

peti commented Apr 10, 2018

LnL7 commented Apr 10, 2018

peti commented Apr 10, 2018

LnL7 commented Apr 10, 2018

peti commented Apr 11, 2018

layus commented Apr 11, 2018

dtzWill commented Apr 11, 2018

layus commented Apr 11, 2018

dtzWill commented Apr 11, 2018

wmertens commented Apr 12, 2018 via email

layus commented Apr 12, 2018

periklis commented May 31, 2018

dtzWill commented May 31, 2018

periklis commented May 31, 2018 via email

kirelagin commented Jun 29, 2018

nixos-discourse commented Feb 6, 2020