Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

haskell/with-packages-wrapper.nix: install "doc" outputs #76842

Merged
merged 2 commits into from Jan 7, 2020
Merged

haskell/with-packages-wrapper.nix: install "doc" outputs #76842

merged 2 commits into from Jan 7, 2020

Conversation

matthewbauer
Copy link
Member

We were previously just installing the "out" output which broke when
we recently changed to generating multiple outputs.

Fixes #76837

Motivation for this change
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
Notify maintainers

cc @

@matthewbauer matthewbauer mentioned this pull request Jan 2, 2020
Closed
@matthewbauer
haskell/with-packages-wrapper.nix: install "doc" outputs
e915608 
We were previously just installing the "out" output which broke when
we recently changed to generating multiple outputs.

Fixes NixOS#76837
@cdepillabout
Copy link
Member

@matthewbauer Could you show me how to test that this actually fixes the problem with hoogle?

My idea was to run a command like the following on the current master:

$ nix-build -E 'with import ./. {}; haskellPackages.ghcWithPackages (p: [p.conduit])'
/nix/store/48grjcgskic4iij38sgdfx4f4q7r0hlr-ghc-8.6.5-with-packages

This works, as expected. In the /nix/store/48grjcgskic4iij38sgdfx4f4q7r0hlr-ghc-8.6.5-with-packages directory, there is no share/doc directory (as expected?).

I then checked out this PR and tried to run the same command, but I got an error:

$ hub checkout https://github.com/NixOS/nixpkgs/pull/76842
$ nix-build -E 'with import ./. {}; haskellPackages.ghcWithPackages (p: [p.conduit])'
these derivations will be built:
  /nix/store/chk26hff7nhhzjyjj5fk80xm3rw3v597-ghc-8.6.3-with-packages.drv
building '/nix/store/chk26hff7nhhzjyjj5fk80xm3rw3v597-ghc-8.6.3-with-packages.drv'...
created 131 symlinks in user environment
rm: cannot remove '/nix/store/0s4vjgsz437aqmpyxxr5zqdcdcdih57j-ghc-8.6.3-with-packages/bin/ghc': Permission denied
builder for '/nix/store/chk26hff7nhhzjyjj5fk80xm3rw3v597-ghc-8.6.3-with-packages.drv' failed with exit code 1
error: build of '/nix/store/chk26hff7nhhzjyjj5fk80xm3rw3v597-ghc-8.6.3-with-packages.drv' failed

I don't understand two things:

  1. Why is this trying to use ghc-8.6.3 (and not the latest version of ghc, 8.6.5)? Is this PR branched off of an old version of nixpkgs? Could you rebase it on top of the current master?
  2. It is probably bad that this is failing, right?

@matthewbauer
Copy link
Member Author

  1. Why is this trying to use ghc-8.6.3 (and not the latest version of ghc, 8.6.5)? Is this PR branched off of an old version of nixpkgs? Could you rebase it on top of the current master?

Yeah this was based off of git merge-base origin/release-19.03 origin/release-19.09 so that it can be merged into master, release-19.09 and release-19.03. It definitely can be rebased, but ideally this fix would be merged into all three branches.

  1. It is probably bad that this is failing, right?

Yeah this is definitely an oversight. It looks like it is a result in moving from symlinkJoin to buildEnv.

@matthewbauer
haskell/with-packages-wrapper.nix: remove /bin symlink if it exists
93aabab 
The wrapper need a writable directory to work, so remove the symlink
to a read-only one if it occurs.
@cdepillabout
Copy link
Member

@matthewbauer

Yeah this was based off of git merge-base origin/release-19.03 origin/release-19.09 so that it can be merged into master, release-19.09 and release-19.03.

Oh, I see. That's really smart! I'm happy with this as-is, then. No need to rebase.

It looks like it is a result in moving from symlinkJoin to buildEnv.

I've tested your fix in 93aabab and this appears to be working for me.

$ nix-build -E 'with import ./. {}; haskellPackages.ghcWithPackages (p: with p; [aeson conduit lens servant-server])'
...
/nix/store/c36g05zmrflibzy5w6wji2zc0nmry3sf-ghc-8.6.3-with-packages
$ ls /nix/store/c36g05zmrflibzy5w6wji2zc0nmry3sf-ghc-8.6.3-with-packages/share/doc/
adjunctions-4.4
aeson-1.4.2.0
ansi-terminal-0.8.2
...

@matthewbauer
Copy link
Member Author

Also it's usually worth trying the GitHub merge commit to test PRs. I think nix-review does this for you, but you can also use something like & test merge_commit_sha directly:

nix-build -E "(import (builtins.fetchTarball \"https://github.com/NixOS/nixpkgs/archive/$(curl https://api.github.com/repos/NixOS/nixpkgs/pulls/76842 | jq -r .merge_commit_sha).tar.gz\") {}).haskellPackages.ghcWithPackages (p: [p.conduit])"

@matthewbauer matthewbauer merged commit ddcaa0c into NixOS:master Jan 7, 2020
@peti peti mentioned this pull request Jan 10, 2020
Merged
sternenseemann added a commit to sternenseemann/nixpkgs that referenced this pull request Jan 14, 2023
@sternenseemann
ghc.withPackages: install documentation to -with-packages output
9733590 
* Will make it so that GHC.Paths's docdir NIX_GHC_DOCDIR points to an
  actual directory.

* Documentation of all packages in the environment is available in
  `$out/share/doc`.

This has previously been attempted in NixOS#76842 and reverted in NixOS#77442,
since documentation can collide when the libraries wouldn't (thanks to
the hash in the lib filename). `symlinkJoin` allows collision, so this
solution should be akin to NixOS#77523 (minus `buildEnv`, one step at a
time).

Collision in the documentation only happen if the dependency closure of
the given packages has more than one different derivation for the same
library of the very same version. I'm personally inclined not to claim
that our infrastructure does anything sensible in this case.
Additionally, the documentation is likely largely the same in such
cases (unless it is heavily patched).

Resolves NixOS#150666.
Resolves NixOS#76837.
Closes NixOS#150968.
Closes NixOS#77523.
@sternenseemann sternenseemann mentioned this pull request Jan 14, 2023
Merged
13 tasks
sternenseemann added a commit to sternenseemann/nixpkgs that referenced this pull request Jan 14, 2023
@sternenseemann
ghc.withPackages: install documentation to -with-packages output
21304d1 
* Will make it so that GHC.Paths's docdir NIX_GHC_DOCDIR points to an
  actual directory.

* Documentation of all packages in the environment is available in
  `$out/share/doc`.

This has previously been attempted in NixOS#76842 and reverted in NixOS#77442,
since documentation can collide when the libraries wouldn't (thanks to
the hash in the lib filename). `symlinkJoin` allows collision, so this
solution should be akin to NixOS#77523 (minus `buildEnv`, one step at a
time). `installDocumentation = false` restores the old behavior.

Collision in the documentation only happen if the dependency closure of
the given packages has more than one different derivation for the same
library of the very same version. I'm personally inclined not to claim
that our infrastructure does anything sensible in this case.
Additionally, the documentation is likely largely the same in such
cases (unless it is heavily patched).

Resolves NixOS#150666.
Resolves NixOS#76837.
Closes NixOS#150968.
Closes NixOS#77523.
sternenseemann added a commit to sternenseemann/nixpkgs that referenced this pull request Jan 15, 2023
@sternenseemann
ghc.withPackages: install documentation to -with-packages output
727491c 
* Will make it so that GHC.Paths's docdir NIX_GHC_DOCDIR points to an
  actual directory.

* Documentation of all packages in the environment is available in
  `$out/share/doc`.

This has previously been attempted in NixOS#76842 and reverted in NixOS#77442,
since documentation can collide when the libraries wouldn't (thanks to
the hash in the lib filename). `symlinkJoin` allows collision, so this
solution should be akin to NixOS#77523 (minus `buildEnv`, one step at a
time). `installDocumentation = false` restores the old behavior.

Collision in the documentation only happen if the dependency closure of
the given packages has more than one different derivation for the same
library of the very same version. I'm personally inclined not to claim
that our infrastructure does anything sensible in this case.
Additionally, the documentation is likely largely the same in such
cases (unless it is heavily patched).

Resolves NixOS#150666.
Resolves NixOS#76837.
Closes NixOS#150968.
Closes NixOS#77523.
lunik1 pushed a commit to lunik1/nixpkgs that referenced this pull request Jan 29, 2023
@sternenseemann @lunik1
ghc.withPackages: install documentation to -with-packages output
0b57105 
* Will make it so that GHC.Paths's docdir NIX_GHC_DOCDIR points to an
  actual directory.

* Documentation of all packages in the environment is available in
  `$out/share/doc`.

This has previously been attempted in NixOS#76842 and reverted in NixOS#77442,
since documentation can collide when the libraries wouldn't (thanks to
the hash in the lib filename). `symlinkJoin` allows collision, so this
solution should be akin to NixOS#77523 (minus `buildEnv`, one step at a
time). `installDocumentation = false` restores the old behavior.

Collision in the documentation only happen if the dependency closure of
the given packages has more than one different derivation for the same
library of the very same version. I'm personally inclined not to claim
that our infrastructure does anything sensible in this case.
Additionally, the documentation is likely largely the same in such
cases (unless it is heavily patched).

Resolves NixOS#150666.
Resolves NixOS#76837.
Closes NixOS#150968.
Closes NixOS#77523.
xanderio pushed a commit to xanderio/nixpkgs that referenced this pull request Feb 13, 2023
@sternenseemann @xanderio
ghc.withPackages: install documentation to -with-packages output
875dc1a 
* Will make it so that GHC.Paths's docdir NIX_GHC_DOCDIR points to an
  actual directory.

* Documentation of all packages in the environment is available in
  `$out/share/doc`.

This has previously been attempted in NixOS#76842 and reverted in NixOS#77442,
since documentation can collide when the libraries wouldn't (thanks to
the hash in the lib filename). `symlinkJoin` allows collision, so this
solution should be akin to NixOS#77523 (minus `buildEnv`, one step at a
time). `installDocumentation = false` restores the old behavior.

Collision in the documentation only happen if the dependency closure of
the given packages has more than one different derivation for the same
library of the very same version. I'm personally inclined not to claim
that our infrastructure does anything sensible in this case.
Additionally, the documentation is likely largely the same in such
cases (unless it is heavily patched).

Resolves NixOS#150666.
Resolves NixOS#76837.
Closes NixOS#150968.
Closes NixOS#77523.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cdepillabout cdepillabout cdepillabout approved these changes

@basvandijk basvandijk Awaiting requested review from basvandijk

@infinisil infinisil Awaiting requested review from infinisil

Assignees
No one assigned
Labels
6.topic: haskell 10.rebuild-darwin: 1-10 10.rebuild-linux: 11-100
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

NIX_GHC_DOCDIR not set properly
3 participants
@matthewbauer @cdepillabout @GuillaumeDesforges