cc @nh2 as this is very relevant for https://github.com/nh2/static-haskell-nix (and anyone interested in using haskell w/musl).
(cc @Ericson2314 on the latter point, IIRC)

Thanks! Merging optimistically and to facilitate testing, feedback appreciated even so :).

Aw bummer the glibc drop-in doesn't quite work on aarch64, might need to be hit with a heftier hammer 😇 .

cc @nh2 as this is very relevant for https://github.com/nh2/static-haskell-nix (and anyone interested in using haskell w/musl).

@dtzWill I don't quite get how I'm supposed to use it.

If I change https://github.com/nh2/static-haskell-nix/blob/9f5351aefda0ddc3c1e8482b4acfbb8077323485/default.nix#L1

to

{ nixpkgs ? import <nixpkgs> { crossSystem = { config = "x86_64-unknown-linux-musl"; }; }, compiler ? "ghc821Binary" }:

and build I get

% NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/2c07921cff84dfb0b9e0f6c2d10ee2bfee6a85ac.tar.gz nix-build --no-out-link
error: assertion failed at /nix/store/w1fndb8swaamrf7amgjjvxfp12nifl2d-2c07921cff84dfb0b9e0f6c2d10ee2bfee6a85ac.tar.gz/pkgs/development/compilers/ghc/8.2.1-binary.nix:7:1

That is the assertion https://github.com/dtzWill/nixpkgs/blob/e065779af185b07566b9bbc4d2fc2ea70fdf3856/pkgs/development/compilers/ghc/8.2.1-binary.nix#L6-L7

@nh2:

1. You are setting crossSystem, for this you want 'localSystem' (hence "native", as opposed to "cross")

2. Recent nixpkgs make this easier to handle, if it's useful, via the 'pkgsMusl' at top-level (and, FWIW, pkgsCross.*).
   So instead of import <nixpkgs> { localSystem.config = "x86_64-unknown-linux-musl"; } you can do (import <nixpkgs> {}).pkgsMusl which is much easier, more discoverable, and doesn't require a different expression to work on other CPU's like aarch64. But you don't need to use it, I'm just very excited it's there and wanted to share :).

@dtzWill Ah, OK! Thanks for these explanations, I really need those. I'm fine with Haskell and linkers but I'm still quite new to how nixpkgs actually does cross-compilation things and what functions are available, so definitely point out when I'm doing things that are more complicated than they should be.

I'm hitting one problem: On your old commit, I did

"--extra-lib-dirs=${pkgs.libiconv.override { enableStatic = true; }}/lib"

this now fails with error: attribute 'override' missing. I'm not sure why that is; I can't locate anything in libiconv that would result in override no longer working.

As mentioned on IRC and such, but posting here for anyone else;

I poked at your repository, here's the sort of thing I ended up with (including answer to your libiconv question):

nh2/static-haskell-nix#3

It's a start!

Wow awesome, thanks a lot! I'll check that out.

@dtzWill This looks ultra promising, with your PR on my project the binary size of the static exe also seems to have gone way down:

Before (https://github.com/nh2/static-haskell-nix/tree/9f5351aefda0ddc3c1e8482b4acfbb8077323485):

14M	/nix/store/0dsc67qhb1s5sa3f0rkmvan5ngqmazs8-blank-me-up-0.1.0.0-x86_64-unknown-linux-musl/bin/blank-me-up

After (your PR):

2.7M	/nix/store/73fh9133xfq88wl7za24x0z1mwfgw615-blank-me-up-0.1.0.0/bin/blank-me-up

Pretty fantastic size reduction!

I don't know what's the reason; I tried to compare the binaries but unfortunately they are stripped (see also #43506) so I can't easily compare the symbols present, so I have to rebuild the old one again in a non-stripped fashion which will have to wait until tomorrow because I GC'd the musl-GHC.

OK I have investigated this difference now:

I have built the mini scotty app blank-me-up with --disable-executable-stripping, run nm -g blank-me-up | cut '-d ' -f3 | sort, for every line removed everything up to the first underscore (to remove package IDs), and thus produced the following two symbol files, one for my old approach with crossSystem and one for the new approach with pkgsMusl:

• blank-me-up-symbols-crossSystem-musl.txt
• blank-me-up-symbols-pkgsMusl.txt

diffing those shows that with pkgsMusl a lot more symbols are missing from the binary, so it's really only linking in code that is uses.

Totally awesome!

I don't know yet why this difference exists, maybe you can tell me @dtzWill.

I have now also built the same excutable with stack on lts-12.0.

• stack build with ld-options: -static: 15 MB
• stack build without ld-options: -static: 13 MB

When building all dependencies with -split-sections (using rm -r ~/.stack/snapshots/x86_64-linux/lts-12.0 && stack clean --full && stack build "--ghc-options=-split-sections"), I get instead:

• stack -split-sections build with ld-options: -static: 21 MB
• stack -split-sections build without ld-options: -static: 19 MB

(all these exes are stripped)

But with our way, it's only 2.7 MB!

So it seems we have found a way that makes executables significantly smaller than the current standard way of building Haskell (4x smaller in this case)!

@nh2 first of this is pretty great!

I'm a bit confused about your symbol listings though. The first few lines of blank-me-up-symbols-pkgsMusl.txt list:

abort
accept
accept4
_acquire_ptc
_aio_close
allocate
bytes_at_heapoverflow
allocateMightFail
allocatePinned
lock
lock
allocGroup
lock
allocGroupOnNode
lock
sync
sync
allocHashTable

allocLargeChunkOnNode
todo_block
tasks
anyUserHandlers
TextziApparziInput_car_closure

Maybe I don't understand sort right.

Also, your pkgsMusl managed to drop Data.Aeson completely from binary (according to the symbols). It would be quite interesting to see where that went and how we figured out we don't
need it.

Your stack builds I assume are non-muslc? You might want to do a symbol comparison with those as well to see why -split-sections results in an increase of 9mb. (That's almost 2/3rds more than the non -split-sections version!)

Maybe I don't understand sort right.

@angerman This is because of the primitive post-processing that I'm doing to get of the package IDs as mentioned above, simply removing everything before the first underscore.

That turns e.g.

allocBlock_lock
allocBlockOnNode_lock
allocGroup
allocGroup_lock

into

lock
lock
allocGroup
lock

Also, your pkgsMusl managed to drop Data.Aeson completely from binary (according to the symbols). It would be quite interesting to see where that went and how we figured out we don't need it.

The scotty app (Main.hs) doesn't use any JSON functionality.

So I suspect that it's because I don't use any scotty function that eventually calls an aeson function.

If I use this diff:

 main = scotty 3000 $ do
     get "/:word" $ do
-        beam <- param "word"
+        beam <- jsonData
         html $ mconcat ["<h1>Scotty, ", beam, " me up!</h1>"]

then I get some aeson functions, but still only those that are actually needed (131 in this case):
blank-me-up-aeson-symbols-for-jsonData.txt

Your stack builds I assume are non-muslc? You might want to do a symbol comparison with those as well

That is true but to get stack musl builds working, I have to build e.g. on Alpine, where the latest ghc and stack currently seem to have problems.

@nh2 can you upload the binaries somewhere?

can you upload the binaries somewhere?

@angerman

• example-scotty-app-pkgsMusl.tar.gz 3.7 MB
  built via tar cf example-scotty-app-pkgsMusl.tar.gz $(NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/2c07921cff84dfb0b9e0f6c2d10ee2bfee6a85ac.tar.gz nix-build --no-link --arg strip false)/bin/example-scotty-app on this commit
• example-scotty-app-crossSystem.tar.gz 19 MB
  built via tar cf example-scotty-app-crossSystem.tar.gz $(NIX_PATH=nixpkgs=https://github.com/dtzWill/nixpkgs/archive/7048fc71e325c69ddfa62309c0b661b430774eac.tar.gz nix-build --no-link --arg strip false)/bin/example-scotty-app on this commit

These have debugging symbols (--arg strip false) so you can easily compare them.

What do I need to test this on Arch Linux? Earlier this year I used Nix for Haskell dev for about a month, but eventually went back to stack.

I'm releasing a binary app soon, and am very much interested in this drastic code size reduction.

@fosskers Right now you have to build with musl.

Try adding your executable here: https://github.com/nh2/static-haskell-nix/blob/627a8c5f86719531e1b1237aec4f5f85982cbd6c/survey/default.nix

Instructions on https://github.com/nh2/static-haskell-nix/tree/627a8c5f86719531e1b1237aec4f5f85982cbd6c#building-arbitrary-packages

@dtzWill I suspect the approach of using both musl and glibc in the same binary as done by this PR (4a1d311) is breaking down now:

We now get segfaults with this approach: #118731

Why does/did this approach make sense? Isn't it generally disallowed to use the musl and glibc in the same binary, e.g. here:

• https://stackoverflow.com/questions/30151808/load-both-musl-libc-so-and-gcc-libc-so-in-the-same-process

Or should ldd on that ghc show only glibc, and musl is accidentally introduced somehow?

@dtzWill I suspect that your approach here exploitet some accidental ABI compatibility between musl and glibc, and that with the change linked in #118731 (commit in musl v1.2.2), this ABI compatibility is no longer given.

But you know these topics much better than me, so some review of this suspicion would be appreciated :)