Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for js-unknown-ghcjs -- for 19.03 #67359

Merged
merged 4 commits into from Sep 2, 2019

Conversation

angerman
Copy link
Contributor

This adds enough logic to nixpkgs to support the js-unknown-ghcjs triple.

Motivation for this change

Support --target=js-unknown-ghcjs where supported.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
Notify maintainers

cc @Ericson2314


# Ghcjs
ghcjs = {
config = "js-unknown-none-ghcjs";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure about this. While I agree that we need better cross-compilation support for ghcjs, "js-unknown-none-ghcjs" is not a widely agreed upon name. Searching for js-unknown-none-ghcjs only gives me #65589. I was going to recommend using "asmjs-unknown-emscripten" which is used by Rust as well, but then realized that GHCJS is not technically asmjs. In addition, we have no C compiler that is compatible with GHCJS, so things like bootstrapping are not well defined.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's okay to allow this one though. GHCJS is in a pretty unique situation, so there is not that much downside to allowing it on an experimental basis. I would like to get some agreement with @luite and ghcjs maintainers on this though.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suggest using js-unknown-ghcjs though. none-ghcjs doesn't really make sense. The "environment" is supposed to qualify the "os", but that doesn't make sense when your os is none. For instance try this:

$ ./config.sub asmjs-unknown-none-ghcjs
Invalid configuration `asmjs-unknown-none-ghcjs': system `none-ghcjs' not recognized

vs.

$ ./config.sub asmjs-unknown-ghcjs
Invalid configuration `asmjs-unknown-ghcjs': system `ghcjs' not recognized

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@matthewbauer thanks for the review. I agree that js-unknown-ghcjs is better. The primary reason for choosing that is that it's already pretty close to how cabal thinks about ghcjs.

We (that includes @luite) will move ghcjs more towards a proper ghc cross compiler. @Ericson2314 will be able to supply you with some more background knowledge from ICFP I guess.

@luite is working on some emscripten support for ghcjs, so we may eventually interact with emscripten, however that may only work from within cabal anyway due to calling convention needs and as such never really be the same as asmjs targets. If we can make that work eventually we may need to add another target.

Note that this needs to be fixed in Nixpkgs, as we can not locally fix it via overlays, as per #65589.

input-output-hk/haskell.nix's overlay branch (input-output-hk/haskell.nix#210) will allow you to treat ghcjs like a regular cross compiler as in

nix build '(with import ./. { nixpkgs = ../nixpkgs; nixpkgsArgs = { crossSystem = { config = "js-unknown-ghcjs"; }; }; }; (haskell-nix.hackage-package { name = "hello"; version = "1.0.0.2"; })).components.exes.hello'

after removing a few more minor kinks.

@angerman
Copy link
Contributor Author

After some additional lengthy discussion with @luite, we'll try to use asmjs-unknown-emscripten and see if we can make that work.

angerman added a commit to input-output-hk/haskell.nix that referenced this pull request Aug 24, 2019
Using the following expression:
```
with import ./. { nixpkgs = ../nixpkgs; nixpkgsArgs = { crossSystem = { config = "js-unknown-ghcjs"; }; }; };
let Cabal = buildPackages.haskell-nix.hackage-package {
    name = "Cabal"; version = "2.4.1.0";
    modules = [
        { packages.Cabal.patches = [ ./Cabal-install-folder.diff ]; }
    ];
}; in
(haskell-nix.hackage-package {
    name = "hello"; version = "1.0.0.2";
    modules = [
    	    ({config, ... }:{ packages.hello.package.setup-depends = [ Cabal ]; })
    ];}).components.exes.hello
```
in a `test.nix` file. And running
```
nix build -f ./test.nix
```
on it, will produce
```
./result
├── bin
│   └── hello.jsexe
│       ├── all.js
│       ├── all.js.externs
│       ├── index.html
│       ├── lib.js
│       ├── manifest.webapp
│       ├── out.frefs.js
│       ├── out.frefs.json
│       ├── out.js
│       ├── out.stats
│       ├── rts.js
│       └── runmain.js
└── share
    └── doc
        └── x86_64-linux-ghc-8.6.5
            └── hello-1.0.0.2
                └── LICENSE

6 directories, 12 files
```

assuming that `nixpkgs` points to NixOS/nixpkgs#67359.
@Ericson2314
Copy link
Member

Err but aren't we outside the asmjs subset?

@angerman
Copy link
Contributor Author

@Ericson2314, morally I think js-unknown-ghcjs is the most correct. @luite is experimenting with the emscripten toolchain for c parts in libraries. Now we could reach for js-ghcjs-emscripten I believe. @luite made the argument that js is not a wildly accepted architecture in autotools and asmjs is.

When choosing between wasm32/wasm and asmjs, i believe we are closer to asmjs in spirit than to wasm :-/

I’ll have to experiment a bit more with more complex libraries today to get a better understanding of the benefits of each triple.

I welcome any ideas!

@Ericson2314
Copy link
Member

Ericson2314 commented Aug 25, 2019

I think let's do js-unknown-ghcjs or js-unknown-none then. If/when the emscripten part is done we can switch, but this is most correct for the GHCJS that is released today. (I can't say I'm wild about enscrypten is its such a big hack that will definitely increase the number of things to worry about and maintain.) I wouldn't worry about what GNU Config does; I rewrote large swaths of it (http://git.savannah.gnu.org/cgit/config.git/log/?qt=author&q=John+Ericson) and am on good terms with the maintainer wrt upstreaming things.

CC @matthewbauer? I want your opinions before I do anything here.

@angerman
Copy link
Contributor Author

ok, let's do js-unknown-ghcjs for now, and switch (or add more) if we end up having it working. js-unknown-ghcjs at least works for simple haskell packages for me already.

@matthewbauer matthewbauer added the 6.topic: cross-compilation Building packages on a different sort platform than than they will be run on label Aug 28, 2019
@angerman
Copy link
Contributor Author

angerman commented Sep 2, 2019

Alright, can we get this merged then?

@@ -20,7 +20,7 @@ stdenv.mkDerivation rec {
];

postPatch =
lib.optionalString ((stdenv.hostPlatform != stdenv.buildPlatform && stdenv.hostPlatform.libc == "msvcrt") || stdenv.cc.nativeLibc)
lib.optionalString ((stdenv.hostPlatform != stdenv.buildPlatform && stdenv.hostPlatform.libc == "msvcrt") || (stdenv.cc != null && stdenv.cc.nativeLibc))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm going to merge without that for now, but we can get back to it tomorrow / in a few hours.

This adds enough logic to nixpkgs to support the `js-unknown-ghcjs` triple.
@Ericson2314 Ericson2314 changed the title Add support for js-unknown-ghcjs Add support for js-unknown-ghcjs -- for 19.03 Sep 2, 2019
@Ericson2314 Ericson2314 mentioned this pull request Sep 2, 2019
10 tasks
@Ericson2314 Ericson2314 merged commit 3e76b65 into NixOS:release-19.03 Sep 2, 2019
@angerman angerman deleted the feature/js-unknown-ghcjs branch September 2, 2019 06:13
angerman added a commit to input-output-hk/haskell.nix that referenced this pull request Sep 16, 2019
This moves the customizations we did in iohk-nix
into haskell.nix via overlays and config.

Add bootPkgs logic

this moves the nuking of bootPkgs where it belongs. This should eventually still be
removed and replaced by a proper solution, that doesn't require the nuking of
bootPkgs.

Allow us to bootstrap binary ghcs and a cabal-install

With this we can do the following:

```
$ nix repl compiler/old-ghc-nix
nix-repl> :b (let pkgs = import <nixpkgs> {}; in with import ./. {}; import ./compiler/bootstrap/cabal-install.nix { inherit (pkgs) fetchurl stdenv zlib; inherit hackage
; ghc = ghc844; src = pkgs.fetchurl { url = "https://github.com/haskell/cabal/archive/Cabal-v3.0.0.0-rc3.tar.gz"; sha256 = "1zl2mgg8307ykq3v8nmafc6zdhhj1cw7w8ffpap16dsm6
5lbnx33"; }; })
```

which wile it may look daunting, will allow us to bootstrap a
cabal-install with a ghc. From that point onwards, we should be able to
build any hackage package via haskell.nix.

Pass through cabal-install version

Better threading of arguments.

Add bootstrap overlay

Allow alex + happy to be built

This still has a wart: we need nix-tools, and for that we use the ghc865 from
nixpkgs. Ideally we'd build nix-tools against a specific ghc, but then
we'd need a build expression for that.

Make ghcjs work

Building something like this:
```
 nix build '(with import ./. { nixpkgs = ../nixpkgs; nixpkgsArgs = { crossSystem = { config = "js-unknown-ghcjs"; }; }; }; (haskell-nix.hackage-package { name = "hello"; version = "1.0.0.2"; })).components.exes.hello'
```
will now work. Assuming `nixpkgs` has been appropriately patched to support the `js-unknown-ghcjs` triple.

Also: this will need an additional `Cabal` patch, to make `Cabal` understand what it needs to do with: `dist/build/hello/hello: copyFile: does not exist (No such file or directory)`

It needs to learn that `dist/build/hello/hello.jsexe` is what it wants to copy and that that is a directory.  Luckily we do have some code in Cabal already that does this for `.exe` on windows.

Build `js-unknown-ghcjs` packages with haskell.nix

Using the following expression:
```
with import ./. { nixpkgs = ../nixpkgs; nixpkgsArgs = { crossSystem = { config = "js-unknown-ghcjs"; }; }; };
let Cabal = buildPackages.haskell-nix.hackage-package {
    name = "Cabal"; version = "2.4.1.0";
    modules = [
        { packages.Cabal.patches = [ ./Cabal-install-folder.diff ]; }
    ];
}; in
(haskell-nix.hackage-package {
    name = "hello"; version = "1.0.0.2";
    modules = [
    	    ({config, ... }:{ packages.hello.package.setup-depends = [ Cabal ]; })
    ];}).components.exes.hello
```
in a `test.nix` file. And running
```
nix build -f ./test.nix
```
on it, will produce
```
./result
├── bin
│   └── hello.jsexe
│       ├── all.js
│       ├── all.js.externs
│       ├── index.html
│       ├── lib.js
│       ├── manifest.webapp
│       ├── out.frefs.js
│       ├── out.frefs.json
│       ├── out.js
│       ├── out.stats
│       ├── rts.js
│       └── runmain.js
└── share
    └── doc
        └── x86_64-linux-ghc-8.6.5
            └── hello-1.0.0.2
                └── LICENSE

6 directories, 12 files
```

assuming that `nixpkgs` points to NixOS/nixpkgs#67359.
angerman added a commit to input-output-hk/haskell.nix that referenced this pull request Sep 16, 2019
This moves the customizations we did in iohk-nix
into haskell.nix via overlays and config.

Add bootPkgs logic

this moves the nuking of bootPkgs where it belongs. This should eventually still be
removed and replaced by a proper solution, that doesn't require the nuking of
bootPkgs.

Allow us to bootstrap binary ghcs and a cabal-install

With this we can do the following:

```
$ nix repl compiler/old-ghc-nix
nix-repl> :b (let pkgs = import <nixpkgs> {}; in with import ./. {}; import ./compiler/bootstrap/cabal-install.nix { inherit (pkgs) fetchurl stdenv zlib; inherit hackage
; ghc = ghc844; src = pkgs.fetchurl { url = "https://github.com/haskell/cabal/archive/Cabal-v3.0.0.0-rc3.tar.gz"; sha256 = "1zl2mgg8307ykq3v8nmafc6zdhhj1cw7w8ffpap16dsm6
5lbnx33"; }; })
```

which wile it may look daunting, will allow us to bootstrap a
cabal-install with a ghc. From that point onwards, we should be able to
build any hackage package via haskell.nix.

Pass through cabal-install version

Better threading of arguments.

Add bootstrap overlay

Allow alex + happy to be built

This still has a wart: we need nix-tools, and for that we use the ghc865 from
nixpkgs. Ideally we'd build nix-tools against a specific ghc, but then
we'd need a build expression for that.

Make ghcjs work

Building something like this:
```
 nix build '(with import ./. { nixpkgs = ../nixpkgs; nixpkgsArgs = { crossSystem = { config = "js-unknown-ghcjs"; }; }; }; (haskell-nix.hackage-package { name = "hello"; version = "1.0.0.2"; })).components.exes.hello'
```
will now work. Assuming `nixpkgs` has been appropriately patched to support the `js-unknown-ghcjs` triple.

Also: this will need an additional `Cabal` patch, to make `Cabal` understand what it needs to do with: `dist/build/hello/hello: copyFile: does not exist (No such file or directory)`

It needs to learn that `dist/build/hello/hello.jsexe` is what it wants to copy and that that is a directory.  Luckily we do have some code in Cabal already that does this for `.exe` on windows.

Build `js-unknown-ghcjs` packages with haskell.nix

Using the following expression:
```
with import ./. { nixpkgs = ../nixpkgs; nixpkgsArgs = { crossSystem = { config = "js-unknown-ghcjs"; }; }; };
let Cabal = buildPackages.haskell-nix.hackage-package {
    name = "Cabal"; version = "2.4.1.0";
    modules = [
        { packages.Cabal.patches = [ ./Cabal-install-folder.diff ]; }
    ];
}; in
(haskell-nix.hackage-package {
    name = "hello"; version = "1.0.0.2";
    modules = [
    	    ({config, ... }:{ packages.hello.package.setup-depends = [ Cabal ]; })
    ];}).components.exes.hello
```
in a `test.nix` file. And running
```
nix build -f ./test.nix
```
on it, will produce
```
./result
├── bin
│   └── hello.jsexe
│       ├── all.js
│       ├── all.js.externs
│       ├── index.html
│       ├── lib.js
│       ├── manifest.webapp
│       ├── out.frefs.js
│       ├── out.frefs.json
│       ├── out.js
│       ├── out.stats
│       ├── rts.js
│       └── runmain.js
└── share
    └── doc
        └── x86_64-linux-ghc-8.6.5
            └── hello-1.0.0.2
                └── LICENSE

6 directories, 12 files
```

assuming that `nixpkgs` points to NixOS/nixpkgs#67359.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
6.topic: cross-compilation Building packages on a different sort platform than than they will be run on 6.topic: stdenv Standard environment 10.rebuild-darwin: 0 10.rebuild-linux: 0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants