Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC 0083] Common interface package sets #83

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

Conversation

FRidh
Copy link
Member

@FRidh FRidh commented Dec 19, 2020

@FRidh
Copy link
Member Author

FRidh commented Dec 19, 2020

cc @adisbladis @doronbehar @jtojnar @Ericson2314 because of your various involvements with package sets.

@shlevy
Copy link
Member

shlevy commented Dec 19, 2020

Possible interaction with flakes: I'd like to be able to have the functions to build a package set in a flake, as well as the ability to declaratively overlay package sets in my flake config.

@Ma27
Copy link
Member

Ma27 commented Dec 19, 2020

Even though I like flakes, I'm not so sure if it's a good idea to design any RFC around any feature that's not yet stabilized (i.e. the APIs can all change and it's not in a stable Nix release) or even documented.

@FRidh FRidh mentioned this pull request Dec 19, 2020
- `fooPackages`
- `foo.pkgs`

TODO which one to pick? Consider also overriding of the interpreter or main program and how that should propagate. Consider also the function for generating variants, where you need to have a name under which your interpreter or main program is available in the subpackage set.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I though we were moving towards foo.pkgs as it makes it harder to have a mismatch of foo versions.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't follow you here. Do you refer to how we pass in say python or pythonPackages instead of the individual Python packages to a non-Python package? That is indeed to prevent mismatches, however, it does not matter whether it is python or pythonPackages.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant that if we have two top-level attributes, overriding just one in an overlay can cause mismatches (unless they refer to each other through scoped names but that has its own downsides). Stuffing everything under the interpreter allows us greater control over references thanks to encapsulation (but maybe at performance cost compared to just bare scopes?).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought we do: fooPackages = lib.recurseIntoAttrs foo.pkgs;.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

recurseIntoAttrs only has effect on whether tools will try to find packages under that attribute. How coupled the two attributes are depends on which scope does foo come from.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've always been wary of foo.pkgs because one compiler/interpreter can build multiple package sets. Having the package set of a tool is philosophically backwards, it should be the compiler/interpreter of the package set.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it should be the compiler/interpreter of the package set

In theory, I would agree. In practice however this makes it so that you have to take care to not pass an incompatible Python version along with it's set.
I think the practicality of this outweighs any philosophical notion.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But in practice both python3.pkgs and python2.pkgs would be the same sets, but with different behavior depending on the python versions which dynamically determine the value of disabled and similar stuff. In that sense it isn't even possible to pass an incompatible python interpreter if I'm not mistaken.

Also in other language-specific package sets there is much less breakage between compiler versions than between python2 and python3 where it would make even less sense.


```nix
fooPackagesFor = foo: import ./foo-packages.nix { ... };
fooPackages_3_6 = fooPackagesFor foo_3_6;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For PHP, we create the php.pkgs internally in the PHP derivation “template” function. That is, a package set for the new variant is just created automatically when the variant is created (e.g. using import "${nixpkgs}/pkgs/development/interpreters/php/generic.nix") { version = "..."; sha256 = "..."; }.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the looks of it is somewhat similar to the passthru for Python.

...


To support nested package sets, the full attribute path is needed, including
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we even want nesting? For example, for PHP, we could have two clearly separated package sets:

  • tools – those are just scripts written in PHP, they would probably not even need to be under PHP, just like we do not put Python applications to python.pkgs.
  • extensions – shared libraries linked against PHP that PHP needs to load at startup

PHP does not really need something like python.withPackages since PHP packages are just text and it is the application, not PHP runtime, responsible for loading them. (AFAIK unlike in Python, anything binary needs to be an extension, not a regular autoloaded package.)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we even want nesting?

I don't know actually. So far it seems we have not, so we can just leave it out of the RFC.

Since then, many other sets also added `withPackages` as part of the
interpreter or main program.

Should the `withPackages` function be part of the main program or the package
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For PHP, withPackages does not make sense either, since we do not actually package any packages other than applications.

For the extensions, I think that it should be php.withExtensions since that does not create an environment but creates a config file listing the extensions and wraps php with a path to the config file.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this shows why withPackages is often enough, but not always and it hence should be a recommendation, but not a rule. In the end, we need a more powerful buildEnv that can compose an environment given a mixture of types of packages using certain rules.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"powerful buildEnv" -> #75 ...

@fare
Copy link

fare commented Dec 19, 2020

In some package sets, you might want to combine multiple aspects: which version of the compiler should I use with which version of the libraries?

Plus some libraries are only available on some versions of the compiler, and I may want to pick and choose: "oh, I'd like these stable libraries with version 3.4 of the compiler, except I want override these particular libraries with newer versions I need—plus their dependencies."

I don't think the job of nixpkgs would be to solve complex sets of dependency constraints, but I believe its job should be to check and enforce the declared constraints (unless explicitly lifted / overridden by the user).

For all these reasons, I think it would be nicer to standardize on some extension mechanism when there are so many incompatible variants of the same thing.

At some point, I would offer POP as a better such extension mechanism to standardize on, that allows for a DAG of dependencies between aspects of your package set. But POP is not quite ready for that yet (the API needs to be made nicer, and I ought to use the C3 algorithm for linearization). [See indeed #82 for a proposal that allows to introduce POP smoothly into nixpkgs, by first tagging it as experimental.]

@fare
Copy link

fare commented Dec 19, 2020

Also, another thing I'm trying to do is allow override of "source" settings before libraries are turned into derivations. Either there should be a standard for such settings to be included in passthru, or for otherwise organizing collections of pre-packages that allow such overrides.

and some additional packages or plugins. This RFC will therefore also recommend
a function each package set should offer for doing so, when appliceable that is.

## Related issues
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Somewhat related NixOS/nixpkgs#75485

packages from the set.

This RFC recommends each package set should have such a function, in case it is
appliceable. An example of where it would not make sense to include such a
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
appliceable. An example of where it would not make sense to include such a
applicable. An example of where it would not make sense to include such a

improvement](https://github.com/NixOS/nixpkgs/pull/105374).

An important aspect for making this work is that, when constructing the package
set, it needs to know its own top-level attribute.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Referring to the incorrect set NixOS/nixpkgs#114538

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet