Non broken fetchcargo #46362

symphorien · 2018-09-08T12:40:19Z

Motivation for this change

This builds on the work of @erictapen #45889
In this comment #45889 (comment) @Mic92 suggested the output of cargo vendor (which is not guaranteed to be stable by upstream) be filtered/normalised to ensure we get a fixed output derivation, somewhat like fetchpatch does.
This is an attempt at implementing it.

What is normalized:

order of key/value pairs
spacing
which quotes are used
the vendoring directory name

what is not normalised

the content of the toml (for example trailing / at the end or urls or which keys appear). Given the current state of cargo vendor, the set of keys which appear is deterministic https://github.com/alexcrichton/cargo-vendor/blob/master/src/main.rs#L48-L67 so I imagine this is fine.

fixes #41518

Things done

Tested that nix-du can build with useRealVendorConfig = true and that I can build the updated habitat
from this PR #46138 without the cargo config hack they are forced to use.

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip" (nothing changes)
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

Mic92 · 2018-09-08T18:04:10Z

doc/languages-frameworks/rust.section.md

@@ -64,6 +65,14 @@ When the `Cargo.lock`, provided by upstream, is not in sync with the
 added in `cargoPatches` will also be prepended to the patches in `patches` at
 build-time.

+The `useRealVendorConfig` parameter tells cargo-vendor to include a Cargo


Do we really have an incompatibility compared to the current state? Is the output different for packages that build at the moment? I would like to enable this by default otherwise.

Otherwise a warning or should be printed otherwise people will never know about this option or bother to migrate.

The incompatibility is: this changes the expected cargoSha256. So two approaches:

we use the new scheme forcibly from now on and break every current hash

or we deprecate the old behavior, let say one nixos release for people to switch to the new behavior, and then remove the old behavior.
This PR implements the second approach.
I agree about the warning.

So you would not just add this:

[source.crates-io] registry = 'https://github.com/rust-lang/crates.io-index' replace-with = 'vendored-sources' [source.vendored-sources] directory = '$(pwd)/$cargoDepsCopy'

to keep the checksum of existing packages in tact and then append the new content as required by the Cargo.toml?

Or is there a problem with this approach?

the right approach is to remove the config when it only contains crates.io. But this is a great idea indeed. I just pushed a commit which should not break any cargoSha256. It is a mass rebuild though. (it rebuilds cargo)

symphorien · 2018-09-09T14:13:33Z

Here is a new commit which does not change any cargosha256 and is thus backward compatible.
I have tried to check that hashes actually don't change, but if reviewers can check it as well this would be nice.

bkchr

LGTM, especially that you save the generated config.

Mic92 · 2018-09-11T20:58:47Z

pkgs/build-support/rust/cargo-vendor-normalise.py

+    return '"{}"'.format(escaped)
+
+
+def main() -> None:


I added type checking here and reformatted the code with black + some minor readability improvements.

Mic92 · 2018-09-11T20:59:24Z

pkgs/applications/misc/alacritty/default.nix

-    fetchSubmodules = true;
+  version = "2018-08-05";
+
+  src = fetchFromGitHub {


I added this as a test, because it makes use of the new .cargo/config

Mic92 · 2018-09-11T21:00:03Z

@GrahamcOfBorg build alacritty

Mic92 · 2018-09-11T21:04:44Z

Can you rebase this on staging?

GrahamcOfBorg · 2018-09-11T21:20:01Z

Unexpected error: command failed with exit code 1 on x86_64-darwin (full log)

Attempted: alacritty

Partial log (click to expand)

   Vendoring ws2_32-sys v0.2.1 (/private/tmp/nix-build-alacritty-unstable-2018-08-05-vendor.drv-0/source/cargo-home.4Ok/registry/src/github.com-1ecc6299db9ec823/ws2_32-sys-0.2.1) to /nix/store/7aw52wn8pcppggy92w5ga12gr46bgz4n-alacritty-unstable-2018-08-05-vendor/ws2_32-sys
   Vendoring x11-dl v2.18.3 (/private/tmp/nix-build-alacritty-unstable-2018-08-05-vendor.drv-0/source/cargo-home.4Ok/registry/src/github.com-1ecc6299db9ec823/x11-dl-2.18.3) to /nix/store/7aw52wn8pcppggy92w5ga12gr46bgz4n-alacritty-unstable-2018-08-05-vendor/x11-dl
   Vendoring xdg v2.1.0 (/private/tmp/nix-build-alacritty-unstable-2018-08-05-vendor.drv-0/source/cargo-home.4Ok/registry/src/github.com-1ecc6299db9ec823/xdg-2.1.0) to /nix/store/7aw52wn8pcppggy92w5ga12gr46bgz4n-alacritty-unstable-2018-08-05-vendor/xdg
   Vendoring xml-rs v0.7.0 (/private/tmp/nix-build-alacritty-unstable-2018-08-05-vendor.drv-0/source/cargo-home.4Ok/registry/src/github.com-1ecc6299db9ec823/xml-rs-0.7.0) to /nix/store/7aw52wn8pcppggy92w5ga12gr46bgz4n-alacritty-unstable-2018-08-05-vendor/xml-rs
   Vendoring yaml-rust v0.4.0 (/private/tmp/nix-build-alacritty-unstable-2018-08-05-vendor.drv-0/source/cargo-home.4Ok/registry/src/github.com-1ecc6299db9ec823/yaml-rust-0.4.0) to /nix/store/7aw52wn8pcppggy92w5ga12gr46bgz4n-alacritty-unstable-2018-08-05-vendor/yaml-rust
To use vendored sources, add this to your .cargo/config for this project:

builder for '/nix/store/7zqjmaaf0k5zl3yz5g9cc7z0x5vhki1j-alacritty-unstable-2018-08-05-vendor.drv' failed with exit code 2
cannot build derivation '/nix/store/x5y8kxq2cxbclphh113hd22g6kvamgmw-alacritty-unstable-2018-08-05.drv': 1 dependencies couldn't be built
error: build of '/nix/store/x5y8kxq2cxbclphh113hd22g6kvamgmw-alacritty-unstable-2018-08-05.drv' failed

GrahamcOfBorg · 2018-09-11T21:21:27Z

Failure on aarch64-linux (full log)

Attempted: alacritty

Partial log (click to expand)

   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ conflicting implementation for `u8`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0119`.
error: Could not compile `winit`.
warning: build failed, waiting for other jobs to finish...
error: build failed
builder for '/nix/store/69pgvahdf92rlg4qsbhifn98wnvzslhc-alacritty-unstable-2018-08-05.drv' failed with exit code 101
error: build of '/nix/store/69pgvahdf92rlg4qsbhifn98wnvzslhc-alacritty-unstable-2018-08-05.drv' failed

Mic92 · 2018-09-11T21:27:50Z

@GrahamcOfBorg build fd

Mic92 · 2018-09-11T21:28:26Z

It would be great if someone with macOS could look at the failure

Mic92 · 2018-09-11T21:52:46Z

@GrahamcOfBorg build fd

Mic92 · 2018-09-11T21:53:02Z

@GrahamcOfBorg build alacritty

Mic92 · 2018-09-11T21:55:23Z

too bad, staging will never finish in time.

GrahamcOfBorg · 2018-09-11T22:08:19Z

Success on x86_64-linux (full log)

Attempted: fd

Partial log (click to expand)

post-installation fixup
moving /nix/store/f6k43h2zp448skjf1252hgndwhlsrz54-fd-7.1.0/man to /nix/store/f6k43h2zp448skjf1252hgndwhlsrz54-fd-7.1.0/share/man
shrinking RPATHs of ELF executables and libraries in /nix/store/f6k43h2zp448skjf1252hgndwhlsrz54-fd-7.1.0
shrinking /nix/store/f6k43h2zp448skjf1252hgndwhlsrz54-fd-7.1.0/bin/fd
gzipping man pages under /nix/store/f6k43h2zp448skjf1252hgndwhlsrz54-fd-7.1.0/share/man/
strip is /nix/store/rf59n87kcrrnrnw3l3s94q879m8vaspm-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/f6k43h2zp448skjf1252hgndwhlsrz54-fd-7.1.0/bin
patching script interpreter paths in /nix/store/f6k43h2zp448skjf1252hgndwhlsrz54-fd-7.1.0
checking for references to /build in /nix/store/f6k43h2zp448skjf1252hgndwhlsrz54-fd-7.1.0...
/nix/store/f6k43h2zp448skjf1252hgndwhlsrz54-fd-7.1.0

GrahamcOfBorg · 2018-09-11T22:21:03Z

Success on x86_64-linux (full log)

Attempted: alacritty

Partial log (click to expand)

post-installation fixup
gzipping man pages under /nix/store/49liwkbackpkin2f3g0xm9iqchwfkk7n-alacritty-unstable-2018-08-05/share/man/
strip is /nix/store/rf59n87kcrrnrnw3l3s94q879m8vaspm-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/49liwkbackpkin2f3g0xm9iqchwfkk7n-alacritty-unstable-2018-08-05/bin
patching script interpreter paths in /nix/store/49liwkbackpkin2f3g0xm9iqchwfkk7n-alacritty-unstable-2018-08-05
checking for references to /build in /nix/store/49liwkbackpkin2f3g0xm9iqchwfkk7n-alacritty-unstable-2018-08-05...
strip is /nix/store/rf59n87kcrrnrnw3l3s94q879m8vaspm-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/xq3rgwcimb9m3w785hljy41nyzhqg8gs-alacritty-unstable-2018-08-05-terminfo
checking for references to /build in /nix/store/xq3rgwcimb9m3w785hljy41nyzhqg8gs-alacritty-unstable-2018-08-05-terminfo...
/nix/store/49liwkbackpkin2f3g0xm9iqchwfkk7n-alacritty-unstable-2018-08-05

LnL7 · 2018-09-11T22:22:43Z

@Mic92 alacritty doesn't build on darwin AFAIK.

GrahamcOfBorg · 2018-09-11T22:40:24Z

Timed out, unknown build status on x86_64-darwin (full log)

Attempted: fd

Partial log (click to expand)

cannot build derivation '/nix/store/ajp1dq347yjpiw53i59cizdd0dw43q90-apple-framework-Security.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/mmspxib9ar4d1mld1zn29hiwlswy6l56-docbook2X-0.8.8.drv': 16 dependencies couldn't be built
cannot build derivation '/nix/store/m14n58jg1mcqg3ywiwz98dgdvi2y7acj-git-2.18.0.drv': 29 dependencies couldn't be built
cannot build derivation '/nix/store/dv0znvjk4syz4255jw06k73sfzbjv9i5-libgit2-0.26.6.drv': 12 dependencies couldn't be built
cannot build derivation '/nix/store/q6srb8vb4bdvi0swwxk73dhi8zs2vwpf-rustc-bootstrap-1.26.2.drv': 6 dependencies couldn't be built
cannot build derivation '/nix/store/l9v5l4fmvzw1d74hax8ysn64cv0cv8ii-cargo-bootstrap-1.26.2.drv': 8 dependencies couldn't be built
cannot build derivation '/nix/store/mi6z331lvzd2iqy8vw88161l3a911r2i-rustc-1.27.0.drv': 15 dependencies couldn't be built
cannot build derivation '/nix/store/g88y3qgqnd0igmmynyj03k4jlbgw8qv1-cargo-1.27.0.drv': 18 dependencies couldn't be built
cannot build derivation '/nix/store/0c5c9wn2809krxcdcjkmaq0rpd9xswlr-fd-7.1.0.drv': 6 dependencies couldn't be built
error: build of '/nix/store/0c5c9wn2809krxcdcjkmaq0rpd9xswlr-fd-7.1.0.drv' failed

GrahamcOfBorg · 2018-09-11T23:07:12Z

Failure on aarch64-linux (full log)

Attempted: fd

Partial log (click to expand)

(cd perl/build/lib && tar cf - .) | \
(cd '/nix/store/slqgvaj54rls5sgsvsq6q5040zkn8fma-git-2.18.0/lib/perl5/site_perl/5.28.0' && umask 022 && tar xof -)
tar: Skipping to next header
tar: Exiting with failure status due to previous errors
make: *** [Makefile:2733: install] Error 2
builder for '/nix/store/819b6207zpq9zigp3rii0lrvvngicl84-git-2.18.0.drv' failed with exit code 2
cannot build derivation '/nix/store/7vgqhdlcx5sgic3w1d7a7si4wz8k1m6q-rustc-1.27.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/3y74nvymh7b0n0vbqcdcsjna4h2yd5jx-cargo-1.27.0.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/jwx0h9p1h7biaci6dj3vmb2h9p0rvhiq-fd-7.1.0.drv': 3 dependencies couldn't be built
error: build of '/nix/store/jwx0h9p1h7biaci6dj3vmb2h9p0rvhiq-fd-7.1.0.drv' failed

GrahamcOfBorg · 2018-09-11T23:11:16Z

Timed out, unknown build status on x86_64-darwin (full log)

Attempted: alacritty

Partial log (click to expand)

cannot build derivation '/nix/store/0fg2mszl1rmld1avjpxndwkqw0jpff11-rust_ignore-0.2.2.drv': 21 dependencies couldn't be built
cannot build derivation '/nix/store/f8yqibkm8i9sy0qfkddg1ixxylws6c1p-apple-framework-AudioToolbox.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/jc0gc3frjqfxq61syvqk1lvpb6s1frd8-mesa-noglu-18.1.8.drv': 32 dependencies couldn't be built
cannot build derivation '/nix/store/yjxpivc3qr37ba1ysmb6z0c4rjrj2phv-rust_cargo-0.22.0.drv': 86 dependencies couldn't be built
cannot build derivation '/nix/store/p25bnbmhmscxnwhh7r7qrl60jkyx9nwy-apple-framework-AppKit.drv': 6 dependencies couldn't be built
cannot build derivation '/nix/store/whdi2yfjx5hkhs8i0c911b6x5lcch64d-libGL-1.0.0.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/213n3kij0pm6qam2n7j11m446pzyi2hw-rust_cargo-vendor-0.1.13.drv': 85 dependencies couldn't be built
cannot build derivation '/nix/store/g4fwpg2qyyk2xxfpxcdpnjxjrhccbm0j-alacritty-unstable-2018-08-05-vendor.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/cxbxcizjra3vb1pd53az4l6vlq7xpb6z-alacritty-unstable-2018-08-05.drv': 29 dependencies couldn't be built
error: build of '/nix/store/cxbxcizjra3vb1pd53az4l6vlq7xpb6z-alacritty-unstable-2018-08-05.drv' failed

GrahamcOfBorg · 2018-09-11T23:18:47Z

Failure on aarch64-linux (full log)

Attempted: alacritty

Partial log (click to expand)

checking for references to /build in /nix/store/amignn5wl2ljwhl953wdr8ladjwdl6mw-mesa-noglu-18.1.8-drivers...
shrinking RPATHs of ELF executables and libraries in /nix/store/zdagshlqwn5idnjdmq9gfyjbyaldza8j-mesa-noglu-18.1.8-osmesa
shrinking /nix/store/zdagshlqwn5idnjdmq9gfyjbyaldza8j-mesa-noglu-18.1.8-osmesa/lib/libOSMesa.so.8.0.0
strip is /nix/store/lw308kxqcm9frx4vlfjh30da1x4gsmwx-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/zdagshlqwn5idnjdmq9gfyjbyaldza8j-mesa-noglu-18.1.8-osmesa/lib
patching script interpreter paths in /nix/store/zdagshlqwn5idnjdmq9gfyjbyaldza8j-mesa-noglu-18.1.8-osmesa
checking for references to /build in /nix/store/zdagshlqwn5idnjdmq9gfyjbyaldza8j-mesa-noglu-18.1.8-osmesa...
building '/nix/store/d1mr1nzmlfwzq8rpmg242hffhyx3141f-libGL-1.0.0.drv'...
cannot build derivation '/nix/store/ls0n3pgsj5pifsq9askwghyd457z52k1-alacritty-unstable-2018-08-05.drv': 3 dependencies couldn't be built
error: build of '/nix/store/ls0n3pgsj5pifsq9askwghyd457z52k1-alacritty-unstable-2018-08-05.drv' failed

Mic92 · 2018-09-12T07:11:17Z

@LnL7 ok, however could you just check if the cargo vendor normalization works for you? ofBorg returned an failure on macOS. Just take any rust package and change the cargoSha256 checksum.

Thanks to NixOS#46362 We can now support git dependencies!

Mic92 · 2018-09-17T18:00:49Z

pkgs/build-support/rust/fetchcargo.nix

+    < $reference $out/bin/cargo-vendor-normalise > test;
+    cmp test $reference
+  '';
+  buildInputs = [ (python3.withPackages(ps: [ ps.toml ])) ];


I just realized we cannot use the results python3.withPackages to replace the shebang of the shell script on macOS, instead we must call the wrapped python executable and pass cargo-vendor-normalise.py as the first argument.
On macOS shebangs must be binaries and cannot be scripts with shebangs itself.

this is what @infinisil got on his machine: https://hastebin.com/uxihohicax.sql

infinisil · 2018-09-17T18:28:31Z

Added a commit that fixes this.

Trying an Alacritty build on Darwin with a slightly changed dependency hash leads to:

To use vendored sources, add this to your .cargo/config for this project:
output path ‘/nix/store/0wqzscwlczmb8s61jc66wwqjpv9milfr-alacritty-unstable-2018-08-05-vendor’ has r:sha256 hash
‘0ms0248bb2lgbzcqks6i0qhn1gaiim3jf1kl17qw52c8an3rc652’ when 
‘1ms0248bb2lgbzcqks6i0qhn1gaiim3jf1kl17qw52c8an3rc652’ was expected
cannot build derivation ‘/nix/store/jkk6kj7vbn2ir9z6rj29liivdzx3zdxy-alacritty-unstable-2018-08-05.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/jkk6kj7vbn2ir9z6rj29liivdzx3zdxy-alacritty-unstable-2018-08-05.drv’ failed

So the hashes are indeed the same :)

I cherry-picked these commits to master so I wouldn't have to build all of stdenv (branch)

infinisil · 2018-09-17T19:01:17Z

Alacritty fully builds on the cherry-picked master branch even :)

Mic92 · 2018-09-17T19:28:26Z

@infinisil thanks!

symphorien · 2018-09-17T19:56:53Z

So this is the "right way" to wrap python scripts...
Thanks!

akru · 2018-12-14T13:14:19Z

@Mic92 dc8aca4 broke this.

Config looked at https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/rust/default.nix#L60 by name config but https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/rust/fetchcargo.nix#L53 install it with name cargo-config.XXX.

symphorien force-pushed the non-broken-fetchcargo branch 2 times, most recently from 4fe2f8f to b5a1b02 Compare September 8, 2018 12:45

GrahamcOfBorg added 6.topic: nixos 6.topic: rust 8.has: changelog 8.has: documentation labels Sep 8, 2018

symphorien mentioned this pull request Sep 8, 2018

buildRustPackage, fetchcargo: optionally use the real config from cargo vendor #45889

Closed

9 tasks

GrahamcOfBorg added 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Sep 8, 2018

Mic92 reviewed Sep 8, 2018

View reviewed changes

GrahamcOfBorg removed 6.topic: nixos 8.has: changelog 8.has: documentation labels Sep 9, 2018

symphorien force-pushed the non-broken-fetchcargo branch from 0246c0e to dcec4dd Compare September 9, 2018 14:17

GrahamcOfBorg added 10.rebuild-darwin: 101-500 10.rebuild-linux: 501+ and removed 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Sep 9, 2018

bkchr approved these changes Sep 10, 2018

View reviewed changes

Mic92 reviewed Sep 11, 2018

View reviewed changes

Mic92 approved these changes Sep 11, 2018

View reviewed changes

Mic92 removed request for Ericson2314 and FRidh September 11, 2018 21:51

infinisil pushed a commit to infinisil/nixpkgs that referenced this pull request Sep 17, 2018

alacritty: switch back to upstream source

2e72c0a

Thanks to NixOS#46362 We can now support git dependencies!

Mic92 reviewed Sep 17, 2018

View reviewed changes

fetchcargo: Fix cargo-vendor-normalise for darwin

50578ab

Mic92 merged commit 8a94866 into NixOS:staging Sep 17, 2018

jtojnar mentioned this pull request Sep 17, 2018

Request: Update alacritty to 0.2.0 #46807

Closed

ljli mentioned this pull request Sep 23, 2018

buildRustPackage: found duplicate version of package #30742

Open

Mic92 mentioned this pull request Sep 25, 2018

habitat: 0.32.2 -> 0.62.0 #46138

Closed

9 tasks

Mic92 mentioned this pull request Oct 28, 2018

Add generated documentation for library functions #49275

Closed

Mic92 mentioned this pull request Dec 8, 2018

parity: 2.1.6 -> 2.1.9; parity-beta: 2.2.1 -> 2.2.4 #51568

Merged

10 tasks

akru mentioned this pull request Dec 8, 2018

polkadot: init at 0.2.17 #51569

Merged

10 tasks

akru mentioned this pull request Dec 14, 2018

fetchcargo: fixed config installation path #52046

Merged

10 tasks

symphorien deleted the non-broken-fetchcargo branch May 18, 2019 16:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Non broken fetchcargo #46362

Non broken fetchcargo #46362

symphorien commented Sep 8, 2018

Mic92 Sep 8, 2018

Mic92 Sep 8, 2018

symphorien Sep 8, 2018

Mic92 Sep 8, 2018

Mic92 Sep 8, 2018

symphorien Sep 9, 2018

symphorien commented Sep 9, 2018

bkchr left a comment

Mic92 Sep 11, 2018

Mic92 Sep 11, 2018

Mic92 commented Sep 11, 2018

Mic92 commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

Mic92 commented Sep 11, 2018

Mic92 commented Sep 11, 2018

Mic92 commented Sep 11, 2018

Mic92 commented Sep 11, 2018

Mic92 commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

LnL7 commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

Mic92 commented Sep 12, 2018

Mic92 Sep 17, 2018

Mic92 Sep 17, 2018

infinisil commented Sep 17, 2018

infinisil commented Sep 17, 2018

Mic92 commented Sep 17, 2018

symphorien commented Sep 17, 2018

akru commented Dec 14, 2018

Non broken fetchcargo #46362

Non broken fetchcargo #46362

Conversation

symphorien commented Sep 8, 2018

Motivation for this change

Things done

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

symphorien commented Sep 9, 2018

bkchr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mic92 commented Sep 11, 2018

Mic92 commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

Mic92 commented Sep 11, 2018

Mic92 commented Sep 11, 2018

Mic92 commented Sep 11, 2018

Mic92 commented Sep 11, 2018

Mic92 commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

LnL7 commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

GrahamcOfBorg commented Sep 11, 2018

Mic92 commented Sep 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

infinisil commented Sep 17, 2018

infinisil commented Sep 17, 2018

Mic92 commented Sep 17, 2018

symphorien commented Sep 17, 2018

akru commented Dec 14, 2018