Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Nixpkgs support for Dhall #79900

Merged
merged 3 commits into from Feb 25, 2020

Conversation

Gabriella439
Copy link
Contributor

One of the motivations for this change is the following Discourse
discussion:

https://discourse.dhall-lang.org/t/offline-use-of-prelude/137

Many users have requested Dhall support for "offline" packages
that can be fetched/built/installed using ordinary package
management tools (like Nix) instead of using Dhall's HTTP import system.
I will continue to use the term "offline" to mean Dhall package
builds that do not use Dhall's language support for HTTP imports (and
instead use the package manager's support for HTTP requests, such
as pkgs.fetchFromGitHub)

The goal of this change is to document what is the idiomatic way to
implement "offline" Dhall builds by implementing Nixpkgs support
for such builds. That way when other package management tools ask
me how to package Dhall with their tools I can refer them to how it
is done in Nixpkgs.

This change contains a fully "offline" build for the largest Dhall
package in existence, known as "dhall-packages" (not to be confused
with dhallPackages, which is our Nix attribute set containing
Dhall packages).

The trick to implementing offline builds in Dhall is to take
advantage of Dhall's support for semantic integrity checks. If an
HTTP import is protected by an integrity check and a cached build
product matches the integrity check then the HTTP import is never
resolved and the expression is instead fetched from cache.

By "installing" dependencies in a pre-seeded and isolated cache
we can replace remote HTTP imports with dependencies that have
been built and supplied by Nix instead.

The offline nature of the builds are enforced by compiling the
Haskell interpreter with the -f-with-http flag, which disables
the interpreter's support for HTTP imports. If a user forgets
to supply a necessary dependency as a Nix build product then the
build fails informing them that HTTP imports are disabled.

By default, built packages are "binary distributions", containing
just a cache product and a Dhall expression which can be used to
resolve the corresponding cache product.

Users can also optionally enable a "source distribution" of a package
which already includes the equivalent fully-evaluated Dhall code (for
convenience), but this is disabled by default to keep /nix/store
utilization as compact as possible.

One of the motivations for this change is the following Discourse
discussion:

https://discourse.dhall-lang.org/t/offline-use-of-prelude/137

Many users have requested Dhall support for "offline" packages
that can be fetched/built/installed using ordinary package
management tools (like Nix) instead of using Dhall's HTTP import system.
I will continue to use the term "offline" to mean Dhall package
builds that do not use Dhall's language support for HTTP imports (and
instead use the package manager's support for HTTP requests, such
as `pkgs.fetchFromGitHub`)

The goal of this change is to document what is the idiomatic way to
implement "offline" Dhall builds by implementing Nixpkgs support
for such builds.  That way when other package management tools ask
me how to package Dhall with their tools I can refer them to how it
is done in Nixpkgs.

This change contains a fully "offline" build for the largest Dhall
package in existence, known as "dhall-packages" (not to be confused
with `dhallPackages`, which is our Nix attribute set containing
Dhall packages).

The trick to implementing offline builds in Dhall is to take
advantage of Dhall's support for semantic integrity checks.  If an
HTTP import is protected by an integrity check and a cached build
product matches the integrity check then the HTTP import is never
resolved and the expression is instead fetched from cache.

By "installing" dependencies in a pre-seeded and isolated cache
we can replace remote HTTP imports with dependencies that have
been built and supplied by Nix instead.

The offline nature of the builds are enforced by compiling the
Haskell interpreter with the `-f-with-http` flag, which disables
the interpreter's support for HTTP imports.  If a user forgets
to supply a necessary dependency as a Nix build product then the
build fails informing them that HTTP imports are disabled.

By default, built packages are "binary distributions", containing
just a cache product and a Dhall expression which can be used to
resolve the corresponding cache product.

Users can also optionally enable a "source distribution" of a package
which already includes the equivalent fully-evaluated Dhall code (for
convenience), but this is disabled by default to keep `/nix/store`
utilization as compact as possible.
There is no need to provide a separate `kubernetesVersion` argument
since the `file` argument works just fine
lib.mapAttrs makePrelude {
# Prelude versions older than 7.0.0 use old-style union literals, which are
# no longer supported by the latest version of the standard
"7.0.0" = {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need all those versions?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Mic92: Not really. I just added as many as possible because it was easy to do. I can remove them if desired

@Profpatsch
Copy link
Member

Will review tomorrow.

Copy link
Member

@Profpatsch Profpatsch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven’t done a full review, but it looks to me like this was mostly inspired by the haskellPackages code, which is a horribly clunky thing in my opinion. I’d advise to keep it more simplistic.

pkgs/development/dhall-modules/Prelude.nix Outdated Show resolved Hide resolved
callPackage ../development/dhall-modules/dhall-kubernetes.nix { };

dhall-packages =
callPackage ../development/dhall-modules/dhall-packages.nix { };
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what’s the reasoning behind this attribute? It doesn’t seem to contain anything that the others don’t expose.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dhall-packages does not just re-export other Dhall packages. dhall-packages provides new functionality of its own (like Argo CD bindings, for example)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I see, dhall-packages is a (mono)repository containing other dhall packages. The naming is kind of confusing.

let
packages = self:
let
callPackage = newScope self;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a fan of the complexity introduced by this scoping override (not a fan of how haskellPackages overcomplicates many things in general). Can we keep it simple? let is a great idea.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Profpatsch: Actually, I was following how pythonPackages did things (which happens to be the same as haskellPackages)

I'm not exactly sure what alternative approach you are suggesting, but it sounds like you are recommending that I use pkgs.callPackage (with no scoping override). I gave that a try and ran into the following issues:

  • It's now more difficult for one Dhall package to depend upon another Dhall package

    ... since the non-overriden callPackage doesn't bring any Dhall packages in scope

  • You lose the ability to override buildDhallPackage

    ... which is really useful, for the same reason that it's useful to override pkgs.haskellPackage.mkDerivation. For example, you could globally override buildDhallPackage to set source = true; to build source distributions for Dhall packages

Or did I misunderstand what you are requesting? Perhaps if you included a short code sample of what you had in mind it would be more clear

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m sorry, I was in a bad mood yesterday and did this review against my better judgement. I should do better than that.

We should use the approach in this PR because of two reasons:

  1. It’s the default, with precedence set by haskellPackages and pythonPackages. There’s a lot of value in regularity, even if the usability or design is not the greatest.
  2. There hasn’t been a better approach so far.

(Prelude."12.0.0".override { file = "Map/Type"; })
];
};
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if dhall-packages (confusing name!) is a monorepository providing vendored versions of other packages, this marks a shortcoming of the proposed approach; how would I as a nix maintainer keep these definitions up to date?
Do I have to rummage through the repository to find all dependencies and manually resolve their version to patch them here? Can dhall help me with this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Profpatsch: My plan was to follow this up with a dhall-to-nixpkgs utility, which would play a role similar to other *2nix utilities like cabal2nix. You would give it a source directory or a URL as input and it would translate that to a derivation matching the style as those underneath ./pkgs/development/dhall-modules

Essentially the tool would:

  • Guess the name of the Dhall package from the directory name or URL
  • Obtain all external dependencies by scanning transitive relative imports
  • Turn those external dependencies into the dependencies field of buildDhallPackage (using the same rule for inferring package names)

For example, if it were to see an external dependency of the form:

https://raw.githubusercontent.com/${user}/${repo}/${version}/${path}

Then it would add this to the dependencies list:

  (repo."${version}".override { file = path })

... or we could include the user in the package name, too, in order to minimize conflicts.

Then the next logical step would be writing a callDhall2Nixpkgs analogous to callCabal2nix

Copy link
Member

@Profpatsch Profpatsch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before I block this any further, let’s merge.

I’m sure you are going to come up with a satisfactory solution iteratively, like you always seem to do 👍

@Profpatsch Profpatsch merged commit 8246c35 into NixOS:master Feb 25, 2020
dtzWill pushed a commit to dtzWill/nixpkgs that referenced this pull request Feb 26, 2020
Add Nixpkgs support for Dhall

(cherry picked from commit 8246c35)
@Gabriella439 Gabriella439 deleted the gabriel/dhall_packages_2 branch February 26, 2020 04:05
@ehmry
Copy link
Contributor

ehmry commented Mar 17, 2020

How does this work without any setup or shell hooks? Why merge if it isn't used anywhere in the tree?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants