haskell: remove unused LDFLAGS in generic-builder #43825

matthewbauer · 2018-07-19T19:03:23Z

Once haskell has found a package.conf.d file, it no longer needs the
traditional -L flag. This commit removes the additional arguments from
NIX_LDFLAGS to prevent some systems from hitting the ARG_MAX limit.
This should accomplish the same thing as moving the $libdir somewhere
else without some of the breakages that would come with.

I don't have a macOS machine right now so cannot verify that this fixes the original issue. I can verify that it correctly removes the extra args from NIX_LDFLAGS though.

/cc @angerman @peti @Ericson2314 @domenkozar @nh2

matthewbauer · 2018-07-19T19:24:18Z

@GrahamcOfBorg build haskell.packages.ghc821Binary.aeson

GrahamcOfBorg · 2018-07-19T19:24:55Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: haskell.packages.ghc821Binary.aeson

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-07-19T19:33:56Z

Failure on x86_64-linux (full log)

Attempted: haskell.packages.ghc821Binary.aeson

Partial log (click to expand)

[14 of 24] Compiling Data.Aeson.Internal.Time ( Data/Aeson/Internal/Time.hs, dist/build/Data/Aeson/Internal/Time.o )
[15 of 24] Compiling Data.Aeson.Encoding.Builder ( Data/Aeson/Encoding/Builder.hs, dist/build/Data/Aeson/Encoding/Builder.o )
[16 of 24] Compiling Data.Aeson.Encoding.Internal ( Data/Aeson/Encoding/Internal.hs, dist/build/Data/Aeson/Encoding/Internal.o )
[17 of 24] Compiling Data.Aeson.Encoding ( Data/Aeson/Encoding.hs, dist/build/Data/Aeson/Encoding.o )
[18 of 24] Compiling Data.Aeson.Types.ToJSON ( Data/Aeson/Types/ToJSON.hs, dist/build/Data/Aeson/Types/ToJSON.o )

<no location info>: error:
    <command line>: can't load .so/.DLL for: libHSprimitive-0.6.3.0-9BHPmR1c3jBGsNX8V107zp.so (libHSprimitive-0.6.3.0-9BHPmR1c3jBGsNX8V107zp.so: cannot open shared object file: No such file or directory)
builder for '/nix/store/hzb8vsgif9znrbivk20layd442cffhqh-aeson-1.3.1.1.drv' failed with exit code 1
error: build of '/nix/store/hzb8vsgif9znrbivk20layd442cffhqh-aeson-1.3.1.1.drv' failed

GrahamcOfBorg · 2018-07-19T19:47:04Z

Failure on x86_64-darwin (full log)

Attempted: haskell.packages.ghc821Binary.aeson

Partial log (click to expand)

    |   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^...
[13 of 24] Compiling Data.Aeson.Internal ( Data/Aeson/Internal.hs, dist/build/Data/Aeson/Internal.o )
[14 of 24] Compiling Data.Aeson.Internal.Time ( Data/Aeson/Internal/Time.hs, dist/build/Data/Aeson/Internal/Time.o )
[15 of 24] Compiling Data.Aeson.Encoding.Builder ( Data/Aeson/Encoding/Builder.hs, dist/build/Data/Aeson/Encoding/Builder.o )
[16 of 24] Compiling Data.Aeson.Encoding.Internal ( Data/Aeson/Encoding/Internal.hs, dist/build/Data/Aeson/Encoding/Internal.o )
[17 of 24] Compiling Data.Aeson.Encoding ( Data/Aeson/Encoding.hs, dist/build/Data/Aeson/Encoding.o )
[18 of 24] Compiling Data.Aeson.Types.ToJSON ( Data/Aeson/Types/ToJSON.hs, dist/build/Data/Aeson/Types/ToJSON.o )
<command line>: can't load .so/.DLL for: libHSprimitive-0.6.3.0-9BHPmR1c3jBGsNX8V107zp.dylib (dlopen(libHSprimitive-0.6.3.0-9BHPmR1c3jBGsNX8V107zp.dylib, 5): image not found)
builder for '/nix/store/01glywapkq5205rql032zmbarwac7wzv-aeson-1.3.1.1.drv' failed with exit code 1
�[31;1merror:�[0m build of '/nix/store/01glywapkq5205rql032zmbarwac7wzv-aeson-1.3.1.1.drv' failed

matthewbauer · 2018-07-19T19:51:48Z

@GrahamcOfBorg build haskellPackages.aeson

GrahamcOfBorg · 2018-07-19T19:52:09Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: haskellPacakages.aeson

Partial log (click to expand)

Cannot nix-instantiate `haskellPacakages.aeson' because:
error: attribute 'haskellPacakages' in selection path 'haskellPacakages.aeson' not found

GrahamcOfBorg · 2018-07-19T19:52:20Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: haskellPackages.aeson

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-07-19T19:52:21Z

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: haskellPacakages.aeson

Partial log (click to expand)

Cannot nix-instantiate `haskellPacakages.aeson' because:
�[31;1merror:�[0m attribute 'haskellPacakages' in selection path 'haskellPacakages.aeson' not found

GrahamcOfBorg · 2018-07-19T19:52:49Z

No attempt on x86_64-linux (full log)

The following builds were skipped because they don't evaluate on x86_64-linux: haskellPacakages.aeson

Partial log (click to expand)

Cannot nix-instantiate `haskellPacakages.aeson' because:
error: attribute 'haskellPacakages' in selection path 'haskellPacakages.aeson' not found

matthewbauer · 2018-07-19T19:56:15Z

@GrahamcOfBorg build haskellPackages.aeson

GrahamcOfBorg · 2018-07-19T19:56:37Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: haskellPackages.aeson

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

matthewbauer · 2018-07-19T20:04:58Z

haskell.packages.ghc821Binary.aeson is broken in master apparently.

GrahamcOfBorg · 2018-07-19T20:23:35Z

Failure on x86_64-linux (full log)

Attempted: haskellPackages.aeson

Partial log (click to expand)

cannot build derivation '/nix/store/11c7ld4ky104pbic8cml42gzvnrmlws9-tasty-quickcheck-0.10.drv': 9 dependencies couldn't be built
cannot build derivation '/nix/store/yhqzyz5w2zymvh37yx3f1ms7ydqy17d8-base-orphans-0.7.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/r7dqa6kxv4gh9j9zgkazmd8z5ipj4awd-generic-deriving-1.12.2.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/ahpp96yyjdfxdz2qslq2czn9nwm67jag-uuid-types-1.0.3.drv': 10 dependencies couldn't be built
cannot build derivation '/nix/store/arzpmycwc4kba5idq880mfsnkrhlrry5-tasty-ant-xml-1.1.4.drv': 6 dependencies couldn't be built
cannot build derivation '/nix/store/rv8xk8wl7vmx4c7y1mjypr7q3bawlhlc-scientific-0.3.6.2.drv': 12 dependencies couldn't be built
cannot build derivation '/nix/store/yj689hd526j91xaz1xgf9d0414ylgfv8-attoparsec-0.13.2.2.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/rmz0jrdi10aknmdxw70h9bh88rlli6hk-quickcheck-instances-0.3.18.drv': 13 dependencies couldn't be built
cannot build derivation '/nix/store/833as91vmwjv3gqya9invsvrni84ay2n-aeson-1.3.1.1.drv': 25 dependencies couldn't be built
error: build of '/nix/store/833as91vmwjv3gqya9invsvrni84ay2n-aeson-1.3.1.1.drv' failed

GrahamcOfBorg · 2018-07-19T20:53:18Z

Failure on x86_64-darwin (full log)

Attempted: haskellPackages.aeson

Partial log (click to expand)

cannot build derivation '/nix/store/y1wkhgkgnssk5864fv7vvjwry6c29alc-tasty-quickcheck-0.10.drv': 9 dependencies couldn't be built
cannot build derivation '/nix/store/9fs67lhn1735an8w74x6b5y55m4gx70b-base-orphans-0.7.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/6zz43lqpng19qj1439izzd8sp30psabd-generic-deriving-1.12.2.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/cxkyag0sgv0qbbg2nkwpy2nl9nn8fb6w-uuid-types-1.0.3.drv': 10 dependencies couldn't be built
cannot build derivation '/nix/store/613g1y5wrl6341f46x0imljhvs4lsf7l-tasty-ant-xml-1.1.4.drv': 6 dependencies couldn't be built
cannot build derivation '/nix/store/s7mmzgk24dpyb2ll7rbfpfqfvw8lhcza-scientific-0.3.6.2.drv': 12 dependencies couldn't be built
cannot build derivation '/nix/store/by703w7aqvx90n5g06cakcwj8qacj821-attoparsec-0.13.2.2.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/9hvjpzyi25i4zpizx0rfzrz528qliaia-quickcheck-instances-0.3.18.drv': 13 dependencies couldn't be built
cannot build derivation '/nix/store/q3m6rdqjjbbsxzsxb7vi9vqdk8k9a52z-aeson-1.3.1.1.drv': 25 dependencies couldn't be built
�[31;1merror:�[0m build of '/nix/store/q3m6rdqjjbbsxzsxb7vi9vqdk8k9a52z-aeson-1.3.1.1.drv' failed

GrahamcOfBorg · 2018-07-19T21:01:22Z

Failure on x86_64-linux (full log)

Attempted: haskellPackages.aeson

Partial log (click to expand)

cannot build derivation '/nix/store/11c7ld4ky104pbic8cml42gzvnrmlws9-tasty-quickcheck-0.10.drv': 9 dependencies couldn't be built
cannot build derivation '/nix/store/yhqzyz5w2zymvh37yx3f1ms7ydqy17d8-base-orphans-0.7.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/r7dqa6kxv4gh9j9zgkazmd8z5ipj4awd-generic-deriving-1.12.2.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/ahpp96yyjdfxdz2qslq2czn9nwm67jag-uuid-types-1.0.3.drv': 10 dependencies couldn't be built
cannot build derivation '/nix/store/arzpmycwc4kba5idq880mfsnkrhlrry5-tasty-ant-xml-1.1.4.drv': 6 dependencies couldn't be built
cannot build derivation '/nix/store/rv8xk8wl7vmx4c7y1mjypr7q3bawlhlc-scientific-0.3.6.2.drv': 12 dependencies couldn't be built
cannot build derivation '/nix/store/yj689hd526j91xaz1xgf9d0414ylgfv8-attoparsec-0.13.2.2.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/rmz0jrdi10aknmdxw70h9bh88rlli6hk-quickcheck-instances-0.3.18.drv': 13 dependencies couldn't be built
cannot build derivation '/nix/store/833as91vmwjv3gqya9invsvrni84ay2n-aeson-1.3.1.1.drv': 25 dependencies couldn't be built
error: build of '/nix/store/833as91vmwjv3gqya9invsvrni84ay2n-aeson-1.3.1.1.drv' failed

GrahamcOfBorg · 2018-07-19T21:53:07Z

Failure on x86_64-darwin (full log)

Attempted: haskellPackages.aeson

Partial log (click to expand)

cannot build derivation '/nix/store/y1wkhgkgnssk5864fv7vvjwry6c29alc-tasty-quickcheck-0.10.drv': 9 dependencies couldn't be built
cannot build derivation '/nix/store/9fs67lhn1735an8w74x6b5y55m4gx70b-base-orphans-0.7.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/6zz43lqpng19qj1439izzd8sp30psabd-generic-deriving-1.12.2.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/cxkyag0sgv0qbbg2nkwpy2nl9nn8fb6w-uuid-types-1.0.3.drv': 10 dependencies couldn't be built
cannot build derivation '/nix/store/613g1y5wrl6341f46x0imljhvs4lsf7l-tasty-ant-xml-1.1.4.drv': 6 dependencies couldn't be built
cannot build derivation '/nix/store/s7mmzgk24dpyb2ll7rbfpfqfvw8lhcza-scientific-0.3.6.2.drv': 12 dependencies couldn't be built
cannot build derivation '/nix/store/by703w7aqvx90n5g06cakcwj8qacj821-attoparsec-0.13.2.2.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/9hvjpzyi25i4zpizx0rfzrz528qliaia-quickcheck-instances-0.3.18.drv': 13 dependencies couldn't be built
cannot build derivation '/nix/store/q3m6rdqjjbbsxzsxb7vi9vqdk8k9a52z-aeson-1.3.1.1.drv': 25 dependencies couldn't be built
�[31;1merror:�[0m build of '/nix/store/q3m6rdqjjbbsxzsxb7vi9vqdk8k9a52z-aeson-1.3.1.1.drv' failed

Ericson2314

Yes I approve of this until we do the mass-rebuild

dezgeg · 2018-07-20T14:45:54Z

Couldn't some build script in principle compile and run some C code that say, generates numeric tables that get embedded in the source code or whatever? Sounds like #43713 achieves the same thing as this PR except breaking such cases.

matthewbauer · 2018-07-20T19:21:39Z

Couldn't some build script in principle compile and run some C code that say, generates numeric tables that get embedded in the source code or whatever? Sounds like #43713 achieves the same thing as this PR except breaking such cases.

I'm a little bit confused because I think the effect of these two PRs should be the same.

I think you're right that in theory a package could have something installed in $out/lib/lib* & want it. But GHC's package.conf.d stuff provides a way to configure the location of these libraries. They shouldn't be relying on LDFLAGS anyway (they can set library locations with library-dirs). I think we want GHC+Cabal to handle as much here as possible.

@angerman

Once haskell has found a package.conf.d file, it no longer needs the traditional -L flag. This commit removes the additional arguments from NIX_LDFLAGS to prevent some systems from hitting the ARG_MAX limit. This should accomplish the same thing as moving the $libdir somewhere else without some of the breakages that would come with /cc @angerman @peti @Ericson2314 - also remove NIX_BUILD_LDFLAGS - protect against accidental removal of added flags in the generic-builder, we want to make sure the NIX_LDFLAGS removal hack only removes the flags that have been added. We make sure that for instance the pattern: -L/nix/store/...-hello-1.0/lib Doesn’t also remove -L/nix/store/...-hello-1.0/lib64 Which must have been added by someone manually (or gotten by some other setup hook).

dezgeg · 2018-07-21T07:16:31Z

No, I mean that a Haskell package declares a dependency on some C library, then at build time compiles and runs some standalone C code that requires that library. With this solution the C program wouldn't link.

domenkozar · 2018-07-21T10:50:34Z

I don't like this approach as well, because it builds on hacks of hacks. I'd rather see stdenv rebuild where we make binutils setup-hook idempotent. I'm going to give that a go.

dezgeg · 2018-07-21T11:05:40Z

Yes I agree, but I think the idea was to have a temporary workaround that isn't a mass rebuild to have Haskell-on-Darwin working sooner. But if you can stand a mass rebuild, I think what I proposed here matthewbauer@110b5e0#r29781349 could do. It's a legit optimization as well to avoid GCC from searching directories that can't possibly contain libraries but might fix Haskell for now.

Of course the duplicate LDFLAGS issue needs fixing regardless.

domenkozar · 2018-07-21T11:09:19Z

If I understand that commit correctly it goes into the right direction, but we want to set LDFLAGS once and then skip the rest of invocations. strictDeps fixes that in general case of setup-hooks, but we could only fix it for binutils setup-hook.

dezgeg · 2018-07-21T11:12:30Z

No, what I propose does nothing to the duplicate LDFLAGS problem, it's a totally orthogonal thing. Instead, it skips adding LDFLAGS for derivations that don't contain any 'real' libraries in $out/lib, which is the case for Haskell and Python (and probably other interpreted language) packages.

domenkozar · 2018-07-21T11:27:10Z

That's a good optimization, given that lib prefix is always required in general - I thought it's a convention, but maybe we can rebuild nixpkgs and see? Maybe we should allow an escape hatch to override skipping?

@dezgeg I think we can get through staging quickly if we pick a solution that doesn't require lots of overrides.

domenkozar · 2018-07-21T11:28:29Z

Actually this won't work as you have lib prefix in haskell libs: /nix/store/566jym0vydcdmpz0li5azwh5cxj36hj6-mmorph-1.1.2/lib/ghc-8.4.3/mmorph-1.1.2/libHSmmorph-1.1.2-u5n0PR0ucK5U9RRLpSUlt.

If you limit yourself to depth 1, lots of packages will break since you have $name/lib/$name/libfoo.hs as a common pattern as well.

dezgeg · 2018-07-21T11:48:17Z

If you limit yourself to depth 1, lots of packages will break [...]

No, it won't break. If you have some package creating $foo/lib/foo/libfoo.so, then a linker flag -L $foo/lib cannot possibly be used to reach that .so file. gcc -lfoo won't work since that would look for $foo/lib/libfoo.so. And gcc -lfoo/foo won't work either since GCC will look for $foo/lib/libfoo/foo.so.

Though now I realize that while this should always work for Linux, I have no idea about other platforms.

angerman · 2018-07-21T11:52:13Z

Linux/BSD/macOS will likely behave quite similar and I think they adhere to the lib... convention, except for frameworks on macOS, but that’s a whole different story with different linker flags and all.

Windows is a completely different beast though. And also often sticks the .dlls into bin. I would just ignore windows for now until we have a unified approach for windows.

domenkozar · 2018-07-21T12:02:56Z

@dezgeg happy to throw 10 hetzner machines to the problem if you code that up :)

dezgeg · 2018-07-21T13:29:34Z

I looked around a bit:

Slashes in -l arguments (like -lfoo/bar) are already unportable; ld.gold does not support them. I'm not really sure how popular ld.gold actually is, however.
Apparently, it's possible to skip the lib prefix with a syntax like gcc -l:foo.a. I have never heard of that syntax, and it's not even documented on gcc manpage, you have to look in ld manpage. And this syntax isn't supported by lld.ld on Darwin.
However with ld.lld on Darwin, using an .o suffix like -lfoo.o will search for a file named foo.o. I can't make enough sense of the binutils code to see if it happens with ld.bfd or ld.gold on Darwin... Which one does macOS use nowadays anyway? And what does Nixpkgs use there?

So, I guess there still could be a theoretical problem with this, but it would have to be quite unconventional code. I guess it's worth giving a try. I pushed an attempt at: https://github.com/dezgeg/nixpkgs/tree/binutils-lflags

matthewbauer · 2018-07-21T13:43:50Z

No, I mean that a Haskell package declares a dependency on some C library, then at build time compiles and runs some standalone C code that requires that library. With this solution the C program wouldn't link.

Well this only removes ldflags it if it has the package.conf.d directory. I guess it's possible for there to be a c library that also has a package.conf.d directory but it seems pretty unlikely (also unsupported by things like stack where I don't think they mess with ldflags at all).

I definitely agree that this is a hack but it seems much safer than moving $out/lib to $out while accomplishing the same thing. As I mentioned there, moving $out/lib will break the withPackages wrapper unless we have some sort of a workaround ready for the moved libdir.

angerman · 2018-07-21T14:13:24Z

@dezgeg macOS has its own linker ld64.

domenkozar · 2018-07-21T18:36:33Z

@dezgeg great, 10 Hetzner servers are crunching. I'll also test macos a tiny bit, then we can push to staging.

domenkozar · 2018-07-21T20:15:16Z

@dezgeg seems to be all good, can you make a PR against staging-next?

domenkozar · 2018-07-21T20:30:52Z

I took the liberty and pushed the commit to staging, release-small.nix builds on linux.

matthewbauer · 2018-07-21T20:59:24Z

Ok awesome! I will close this for now. Hopefully it is no longer needed!

nh2 · 2018-09-10T01:00:24Z

Can somebody update me: Is a fix for this in master by now?

matthewbauer · 2018-09-10T03:07:16Z

Yes it is definitely in master now.

nh2 · 2018-09-10T12:06:24Z

I see, it's commit 24596d7, pulled in with 099c13d.

matthewbauer requested a review from peti as a code owner July 19, 2018 19:03

matthewbauer force-pushed the haskell-remove-ldflags branch from cdf58ae to 9495d52 Compare July 19, 2018 19:03

matthewbauer requested review from domenkozar and Ericson2314 July 19, 2018 19:04

GrahamcOfBorg added 6.topic: haskell 10.rebuild-darwin: 501+ 10.rebuild-linux: 501+ labels Jul 19, 2018

matthewbauer mentioned this pull request Jul 19, 2018

Feature/no more haskell libs #43713

Closed

9 tasks

Ericson2314 approved these changes Jul 19, 2018

View reviewed changes

peti removed their request for review July 20, 2018 16:41

matthewbauer force-pushed the haskell-remove-ldflags branch from f4800de to bd6b945 Compare July 21, 2018 01:09

matthewbauer closed this Jul 21, 2018

haskell: remove unused LDFLAGS in generic-builder #43825

haskell: remove unused LDFLAGS in generic-builder #43825

Conversation

matthewbauer commented Jul 19, 2018 • edited

matthewbauer commented Jul 19, 2018

GrahamcOfBorg commented Jul 19, 2018

GrahamcOfBorg commented Jul 19, 2018

GrahamcOfBorg commented Jul 19, 2018

matthewbauer commented Jul 19, 2018 • edited

GrahamcOfBorg commented Jul 19, 2018

GrahamcOfBorg commented Jul 19, 2018

GrahamcOfBorg commented Jul 19, 2018

GrahamcOfBorg commented Jul 19, 2018

matthewbauer commented Jul 19, 2018

GrahamcOfBorg commented Jul 19, 2018

matthewbauer commented Jul 19, 2018

GrahamcOfBorg commented Jul 19, 2018

GrahamcOfBorg commented Jul 19, 2018

GrahamcOfBorg commented Jul 19, 2018

GrahamcOfBorg commented Jul 19, 2018

Ericson2314 left a comment

Choose a reason for hiding this comment

dezgeg commented Jul 20, 2018

matthewbauer commented Jul 20, 2018 • edited

dezgeg commented Jul 21, 2018

domenkozar commented Jul 21, 2018

dezgeg commented Jul 21, 2018

domenkozar commented Jul 21, 2018

dezgeg commented Jul 21, 2018 • edited

domenkozar commented Jul 21, 2018

domenkozar commented Jul 21, 2018 • edited

dezgeg commented Jul 21, 2018 • edited

angerman commented Jul 21, 2018

domenkozar commented Jul 21, 2018

dezgeg commented Jul 21, 2018

matthewbauer commented Jul 21, 2018 • edited

angerman commented Jul 21, 2018

domenkozar commented Jul 21, 2018

domenkozar commented Jul 21, 2018

domenkozar commented Jul 21, 2018

matthewbauer commented Jul 21, 2018

nh2 commented Sep 10, 2018

matthewbauer commented Sep 10, 2018

nh2 commented Sep 10, 2018

matthewbauer commented Jul 19, 2018 •

edited

matthewbauer commented Jul 19, 2018 •

edited

matthewbauer commented Jul 20, 2018 •

edited

dezgeg commented Jul 21, 2018 •

edited

domenkozar commented Jul 21, 2018 •

edited

dezgeg commented Jul 21, 2018 •

edited

matthewbauer commented Jul 21, 2018 •

edited