Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

racket: on darwin, use libiconv #35429

Merged
merged 2 commits into from Nov 16, 2018
Merged

racket: on darwin, use libiconv #35429

merged 2 commits into from Nov 16, 2018

Conversation

clacke
Copy link
Contributor

@clacke clacke commented Feb 23, 2018

Without libiconv, racket doesn't support UTF-16, and this breaks one
of the tests.

Closes #34576

Motivation for this change

Make racket build on Darwin.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • [] NixOS
    • [✓] macOS
    • [✓] other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • [✓] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • [✓] Tested execution of all binary files (usually in ./result/bin/)
  • [✓] Fits CONTRIBUTING.md.

The binaries that require libX11.6.dylib in the library path for dynamic loading don't run, but I think that's in scope for another commit.

On Linux side this commit introduces zero change to derivations.

Without libiconv, racket doesn't support UTF-16, and this breaks one
of the tests.

Closes NixOS#34576
@Mic92
Copy link
Member

Mic92 commented Feb 24, 2018

@GrahamcOfBorg build racket racket-minima

@Mic92
Copy link
Member

Mic92 commented Feb 24, 2018

@GrahamcOfBorg build racket racket-minimal

@GrahamcOfBorg
Copy link

Failure on aarch64-linux (full log)

Partial log (click to expand)

Package ‘racket-6.12’ in /var/lib/gc-of-borg/nix-test-rs-3/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-3/pkgs/development/interpreters/racket/default.nix:84 is not supported on ‘aarch64-linux’, refusing to evaluate.

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.

@GrahamcOfBorg
Copy link

Failure on aarch64-linux (full log)

Partial log (click to expand)

Package ‘racket-6.12’ in /var/lib/gc-of-borg/nix-test-rs-1/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-1/pkgs/development/interpreters/racket/default.nix:84 is not supported on ‘aarch64-linux’, refusing to evaluate.

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.

@GrahamcOfBorg
Copy link

Failure on x86_64-darwin (full log)

Partial log (click to expand)

error: attribute 'racket-minima' in selection path 'racket-minima' not found

@GrahamcOfBorg
Copy link

Failure on x86_64-linux (full log)

Partial log (click to expand)

error: attribute ‘racket-minima’ in selection path ‘racket-minima’ not found

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Partial log (click to expand)

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0 5315k    0 16073    0     0  61819      0  0:01:28 --:--:--  0:01:28 61582
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 5315k  100 5315k    0     0  5435k      0 --:--:-- --:--:-- --:--:-- 5429k


  9 74.1M    9 7523k    0     0  5569k      0  0:00:13  0:00:01  0:00:12 5565k
 28 74.1M   28 20.9M    0     0  9330k      0  0:00:08  0:00:02  0:00:06 9326k
 45 74.1M   45 33.6M    0     0  10.1M      0  0:00:07  0:00:03  0:00:04 10.1M
 60 74.1M   60 45.1M    0     0  10.4M      0  0:00:07  0:00:04  0:00:03 10.4M
 76 74.1M   76 56.8M    0     0  10.6M      0  0:00:06  0:00:05  0:00:01 11.2M
 92 74.1M   92 68.3M    0     0  10.8M      0  0:00:06  0:00:06 --:--:-- 12.3M
100 74.1M  100 74.1M    0     0  10.8M      0  0:00:06  0:00:06 --:--:-- 11.7M

/nix/store/jvf7fjn1xv8aljyf2gj9671ik7c6dbgf-racket-6.12
/nix/store/0i492hfziqhlmgxcp5z0iyrmbpsbh97p-racket-minimal-6.12

@GrahamcOfBorg
Copy link

Failure on x86_64-darwin (full log)

Partial log (click to expand)

raco setup:      /nix/store/2j30avvbvnahw9k5s0mvhf1cx5w82a5v-racket-6.12/share/racket/pkgs/htdp-lib/2htdp/image.rkt: [traversing imports]
raco setup:      /nix/store/2j30avvbvnahw9k5s0mvhf1cx5w82a5v-racket-6.12/share/racket/collects/racket/private/more-scheme.rkt:148:2: call-with-break-parameterization
raco setup:      .../more-scheme.rkt:261:28
raco setup:      /nix/store/2j30avvbvnahw9k5s0mvhf1cx5w82a5v-racket-6.12/share/racket/pkgs/sandbox-lib/racket/sandbox.rkt:861:5: loop
raco setup:
make[1]: *** [Makefile:178: install-3m] Error 1
make[1]: Leaving directory '/private/tmp/nix-build-racket-6.12.drv-0/racket-6.12/src/build'
make: *** [Makefile:103: install] Error 2
builder for '/nix/store/hhyc9zmrr9jksdv09zdjiz11d0svw9d4-racket-6.12.drv' failed with exit code 2
error: build of '/nix/store/hhyc9zmrr9jksdv09zdjiz11d0svw9d4-racket-6.12.drv' failed

@clacke
Copy link
Contributor Author

clacke commented Feb 24, 2018

Guess I'll try it in sandbox.

@clacke
Copy link
Contributor Author

clacke commented Feb 25, 2018

Oh. What do I do to fix this? I set build-use-sandbox = relaxed and reloaded the nix-daemon service, and now I'm getting:

building path(s) ‘/nix/store/cs447fnh7r560wgqrsc6qxdpg648cmm9-racket-6.12’
/nix/store/0igxxngigjgfkgg3cw547n92b8bpg6b1-stdenv-darwin/setup: fork: Operation not permitted

NixOS/nix#759 seems to say this is a problem that was fixed two years ago. Has it come back or am I doing something wrong?

It may be relevant that nix-info -m claims - multi-user?: no when I'm running with sandbox = relaxed? Or is that normal?

@LnL7
Copy link
Member

LnL7 commented Feb 25, 2018

Sandboxing requires nix 2.0 and the relaxed mode isn't really supported anymore.

@clacke
Copy link
Contributor Author

clacke commented Feb 25, 2018

Ok! Time to upgrade then.

@clacke
Copy link
Contributor Author

clacke commented Feb 26, 2018

That wasn't a roaring success.

configure:3839: ./conftest
dyld: Library not loaded: @rpath/CoreFoundation.framework/Versions/A/CoreFoundation
  Referenced from: /private/tmp/nix-build-racket-6.12.drv-0/racket-6.12/src/build/./conftest
  Reason: no suitable image found.  Did find:
        /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation: file system sandbox blocked stat()

@clacke
Copy link
Contributor Author

clacke commented Feb 26, 2018

@LnL7 It seems to me that

useSystemCoreFoundationFramework () {
export NIX_COREFOUNDATION_RPATH=/System/Library/Frameworks
}
addEnvHooks "$hostOffset" useSystemCoreFoundationFramework
, activated by
# don't use pure CF for dylibs that depend on frameworks
setupHook = ./framework-setup-hook.sh;
, makes sandboxing impossible? Alternatively, sandboxing needs to be opened up to system CF?

Probably

# On macOS, frameworks are linked to the system CoreFoundation but
# dynamic libraries built with nix use a pure version of CF this
# causes segfaults for binaries that depend on it at runtime. This
# can be solved in two ways.
can explain why my first attempt at solving #35452 segfaulted?

@LnL7
Copy link
Member

LnL7 commented Feb 26, 2018

Yeah, there are still some problems with sandboxing and frameworks. FYI the darwin builders don't run with sandboxing enabled anymore.

@Mic92
Copy link
Member

Mic92 commented Feb 27, 2018

I already noticed it, when /bin/bash did not return a failure anymore.

@clacke
Copy link
Contributor Author

clacke commented Feb 27, 2018

Ok. Any suggestions how I can fix this? glib is in the libPath expression, but I noticed that LD_LIBRARY_PATH isn't in the environment when building. And on MacOS maybe we should set DYLD_LIBRARY_PATH instead?

@LnL7
Copy link
Member

LnL7 commented Mar 1, 2018

I think that's equivalent. There are also a bunch of other DYLD_* environment variables described in man dyld that might help when debugging something like this.

clacke added a commit to clacke/fractalide that referenced this pull request Mar 31, 2018
See NixOS/nixpkgs#35429 .

It's not good enough to merge into nixpkgs, but it's good enough for
us.

Solution: On Nix/Darwin, make racket use GNU libiconv.
clacke added a commit to clacke/fractalide that referenced this pull request Mar 31, 2018
See NixOS/nixpkgs#35429 .

It's not good enough to merge into nixpkgs, but it's good enough for
us.

Solution: On Nix/Darwin, make racket use GNU libiconv.
clacke added a commit to clacke/fractalide that referenced this pull request Mar 31, 2018
See NixOS/nixpkgs#35429 .

It's not good enough to merge into nixpkgs, but it's good enough for
us.

Solution: On Nix/Darwin, make racket use GNU libiconv.
clacke added a commit to clacke/fractalide that referenced this pull request Mar 31, 2018
See NixOS/nixpkgs#35429 .

It's not good enough to merge into nixpkgs, but it's good enough to
build.

Solution: On Nix/Darwin, make racket use GNU libiconv.

Now pkgs.fractalide builds.

Known issues:

```
$ $(nix-build --no-out-link -A pkgs.fractalide)/bin/hyperflow
[ . . . ]
ffi-lib: couldn't open "libX11.6.dylib" (dlopen(libX11.6.dylib, 6): image not found)
  context...:
   /nix/store/i4lc5qygg6jzdy5p0fam7s7p700c6agj-racket-6.12/share/racket/pkgs/gui-lib/mred/private/wx/gtk/x11.rkt: [running body]
```

```
$ DYLD_LIBRARY_PATH=$(nix-build --no-out-link -A pkgs.xorg.libX11)/lib:$DYLD_LIBRARY_PATH $(nix-build --no-out-link -A pkgs.fractalide)/bin/hyperflow
[ . . . ]
Segmentation fault: 11
```

```
$ DYLD_LIBRARY_PATH=$(nix-build --no-out-link -A pkgs.xorg.libX11)/lib:$DYLD_LIBRARY_PATH $(nix-build --no-out-link -A pkgs.fractalide)/bin/hyperflow
racket(11288,0x7fffda7df3c0) malloc: *** error for object 0x10e430d68: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6
```

But it builds!
clacke added a commit to clacke/fractalide that referenced this pull request Mar 31, 2018
This will time out every time. If we want to test on darwin, we have a
few options:

 - Solve NixOS/nixpkgs#35429
   Get racket into nixpkgs, so we can pull down racket from hydra.

 - Solve fractalide/racket2nix#94
   Make racket2nix competent enough that we don't get the empty.zip
   sources for core packages, and can build on racket-minimal.

 - Massage the catalogs
   If we make sure that the core packages come from the release
   catalog instead of the live catalog, it should work. This is
   probably as easy as a union between two hashes. We could even
   cheat and just manually add only typed-map to the release
   catalog.

Solution: Disable darwin for now.
@clacke
Copy link
Contributor Author

clacke commented Nov 16, 2018

Re: {DY,}LD_LIBRARY_PATH, I have since learned that Racket has internal variables that the module loader looks at, and that these environment variables don't help.

Full Racket doesn't build on darwin even with libiconv (the fix in this PR), but at least it would help us remove our local patch in racket2nix.

With racket2nix on darwin today, with this fix, we can build racket-minimal plus the drracket package and all its dependencies, and run drracket with a whole bunch of caveats. It's not ready to replace full racket yet.

The next step to make full racket and drracket work on Nix darwin would be to get the racket derivation a config.rktd with the lib-search-dirs setting, like we do for our racket packages:

https://github.com/fractalide/racket2nix/blob/f3a7d10f35d960dfa43f679205b394bb4f9d2c7e/nix/racket2nix.rkt#L161

@sjmackenzie
Copy link
Contributor

@qknight this is mergable, mind taking a quick review please.

@Mic92
Copy link
Member

Mic92 commented Nov 16, 2018

@GrahamcOfBorg build racket racket-minimal

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: racket-minimal

The following builds were skipped because they don't evaluate on aarch64-linux: racket

Partial log (click to expand)



these paths will be fetched (6.37 MiB download, 31.09 MiB unpacked):
  /nix/store/3ipjzjm5hxk9h6sh5c791akpxcpw6gli-libtool-2.4.6
  /nix/store/a351yhxr4l2x3a45x2gflsj3dg2vqmcv-racket-minimal-7.1
  /nix/store/xxjz7zmn5jbvyp7rbcklgx10c23fbqf5-poppler-glib-0.67.0
copying path '/nix/store/3ipjzjm5hxk9h6sh5c791akpxcpw6gli-libtool-2.4.6' from 'https://cache.nixos.org'...
copying path '/nix/store/xxjz7zmn5jbvyp7rbcklgx10c23fbqf5-poppler-glib-0.67.0' from 'https://cache.nixos.org'...
copying path '/nix/store/a351yhxr4l2x3a45x2gflsj3dg2vqmcv-racket-minimal-7.1' from 'https://cache.nixos.org'...
/nix/store/a351yhxr4l2x3a45x2gflsj3dg2vqmcv-racket-minimal-7.1

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: racket, racket-minimal

Partial log (click to expand)

copying path '/nix/store/jnn7kk6rsrpwr8ks80f60d4pkm38crm1-libmicrohttpd-0.9.59' from 'https://nix-cache.s3.amazonaws.com'...
copying path '/nix/store/haf7c8iah82an4r044sbxvlq83yy9ffh-systemd-239' from 'https://nix-cache.s3.amazonaws.com'...
copying path '/nix/store/n0lb5g87hd7xaa09pinaf74r4p9ckgsi-dbus-1.12.10' from 'https://nix-cache.s3.amazonaws.com'...
copying path '/nix/store/xzgmxp8ry1kwbpq8sgm4srg9jmmfb44m-at-spi2-core-2.28.0' from 'https://nix-cache.s3.amazonaws.com'...
copying path '/nix/store/jglz63ny49xcjjkd9240di4p0z05mspc-at-spi2-atk-2.26.2' from 'https://nix-cache.s3.amazonaws.com'...
copying path '/nix/store/p4b0avnpcvvj6b108zba9j61fgir5b2k-gtk+3-3.22.30' from 'https://nix-cache.s3.amazonaws.com'...
copying path '/nix/store/9q8vl7kzmckcrvn5nmirp6s13ysznrxl-racket-7.1' from 'https://nix-cache.s3.amazonaws.com'...
copying path '/nix/store/ymb05vac1gvbiwd1g5dhdicwngr5bhw1-racket-minimal-7.1' from 'https://nix-cache.s3.amazonaws.com'...
/nix/store/9q8vl7kzmckcrvn5nmirp6s13ysznrxl-racket-7.1
/nix/store/ymb05vac1gvbiwd1g5dhdicwngr5bhw1-racket-minimal-7.1

@GrahamcOfBorg
Copy link

Failure on x86_64-darwin (full log)

Attempted: racket, racket-minimal

Partial log (click to expand)

raco setup:      for-loop
raco setup:      [repeats 1 more time]
raco setup:      run-module-instance!125
raco setup:      ...
raco setup:
make[1]: *** [Makefile:177: install-3m] Error 1
make[1]: Leaving directory '/private/tmp/nix-build-racket-7.1.drv-0/racket-7.1/src/build'
make: *** [Makefile:103: install] Error 2
builder for '/nix/store/fx0j7dqisr8yjv0kzdwnhhf4fx5dbc3g-racket-7.1.drv' failed with exit code 2
error: build of '/nix/store/fx0j7dqisr8yjv0kzdwnhhf4fx5dbc3g-racket-7.1.drv' failed

@Mic92 Mic92 merged commit d59307e into NixOS:master Nov 16, 2018
@clacke clacke deleted the racket-full-darwin branch November 21, 2018 14:18
clacke added a commit to clacke/racket2nix-clacke that referenced this pull request Nov 29, 2018
nixpkgs-unstable has darwin racket depending on libiconv as of:

NixOS/nixpkgs#35429 d59307e54c63daf42d98288ba4c7132ccc02daf2

Solution: Remove racket-minimal override.
clacke added a commit to clacke/racket2nix-clacke that referenced this pull request Nov 29, 2018
nixpkgs-unstable has darwin racket depending on libiconv as of:

NixOS/nixpkgs#35429 d59307e54c63daf42d98288ba4c7132ccc02daf2

Solution: Remove racket-minimal override.

Closes fractalide#230
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants