I'm not quite sure about this idiom. Say we had this function:

let
  # injectOverride
  # :  (name: String)
  # -> ({ a } -> { a } -> { a })
  # -> ({ name: {a}; ..b } -> { name: {a}; ..b } -> { name: {a}; ..b })
  injectOverride = name: f: self: super: { 
    "${name}" = super.${name} // f self.${name} super.${self} // {
      __overlay__ = lib.composeExtensions (super.__inner__ or (_: _: {})) f;
    };
  }; 

Then we can just do:

import <nixpkgs> { overlays = [
  # Add an interpreter and its package set.
  (self: super: 
  let 
    pkgs = self;
    interpreterPackages = self: {
      callPackage = pkgs.newScope self;
      foo = self.callPackage ./pkgs/interpreter/foo { /* interpreter libxml */ };
      bar = self.callPackage ./pkgs/interpreter/bar { /* interpreter foo */ };
      baz = self.callPackage ./pkgs/interpreter/baz { /* interpreter foo bar qt */ };
      qux = self.callPackage ./pkgs/interpreter/qux { /* interpreter bar baz */ };
    };
  in
  rec {
    interpreter27 = self.callPackage ./pkgs/interpreter_27.nix {};
    interpreter35 = self.callPackage ./pkgs/interpreter_35.nix {};
  } 
  // injectOverride "interpreter27Packages"
       (lib.extend (_: _: { interpreter = interpreter27; }) interpreterPackages)
  // injectOverride "interpreter35Packages"
       (lib.extend (_: _: { interpreter = interpreter35; }) interpreterPackages))
  # Change a package dependency. Like any ordinary package, as opposed as today.
  (injectOverride "interpreter27Packages" {
      # Will change foo within bar, baz, qux (as expected).
      foo = super.interpreter27Packages.foo.override { libxml = null; };
  })
  # Duplicate a package set with a custom interpreter.
  (self: super: injectOverride "interpreter27ProfilePackages"
    super.interpreter27Packages.__overlay__ self super)
]; }

@Ericson2314
The problem with the idiom you are suggesting is that users of a given sub-set of packages will have to know about the injectOverride function, and know when to use it. As opposed to overrideWithScope which would only be required at the creation/cloning of the sub-set.

(Also this should always be super.callPackage)

Honestly, if post-processing of Nixpkgs was an option, and every package would use callPackage (which is far from being the case), then we could make the callPackage function return a functor like object which is derived into a derivation only after going through the Nixpkgs fix-point, and thus we could go with just the following:

import <nixpkgs> { overlays = [
  # Add an interpreter and its package set.
  (self: super: 
  let interpreterPackages = {
      foo = super.callPackage ./pkgs/interpreter/foo { /* interpreter libxml */ };
      bar = super.callPackage ./pkgs/interpreter/bar { /* interpreter foo */ };
      baz = super.callPackage ./pkgs/interpreter/baz { /* interpreter foo bar qt */ };
      qux = super.callPackage ./pkgs/interpreter/qux { /* interpreter bar baz */ };
    };
  in
  rec {
    interpreter27 = super.callPackage ./pkgs/interpreter_27.nix {};
    interpreter35 = super.callPackage ./pkgs/interpreter_35.nix {};
    interpreter27Packages = { interpreter = interpreter27; } // interpreterPackages;
    interpreter35Packages = { interpreter = interpreter35; } // interpreterPackages;
  })
  # Change a package dependency. Like any ordinary package, as opposed as today.
  (self: super: {
    interpreter27Packages = super.interpreter27Packages // {
      # Will change foo within bar, baz, qux (as expected).
      foo = super.interpreter27Packages.foo.override { libxml = null; };
    };
  })
  # Duplicate a package set with a custom interpreter.
  (self: super: {
    interpreter27ProfilePackages = super.interpreter27Packages // {
      interpreter = super.interpreter27.override { withProfiler = true; };
    });
  })
]; }

As the scope would be determine by the location of the package within Nixpkgs. But this future is not yet there, because not every package uses callPackage, and not every dependency comes from self.

For a bit now, I've been thinking it would be nice if the callPackage mechanism could add an override function that would do the equivalent of adding another overlay and then re-evaluate the package.

The idea would be then be that you could also effect changes in all the dependencies of a package as well. Typical examples would be where the dependent packages must also be changed or else you windup with the final executable sucking in two different versions of the same shared library (you see this in some existing expressions with the version of boost or cuda libraries used in its dependencies).

I originally thought this might be what this pull request was, but, upon closer looking, I don't believe so.
I thought I would mention it here though as it does seem there is some significant overlap.

Thanks! -Tyson

PS: Technically, I expect, it could make sense to only apply the overlay to runtime dependencies (i.e., not to the build time ones).

Can we move forward with this?

Does this need review? testers? redesign?

I now think #51213 is a better approach than this and my first comment's suggestion.

In #54266 I make it even simpler to extend existing package sets, so instead of

  (self: super: {
    interpreter27Packages = super.interpreter27Packages // {
      # Will change foo within bar, baz, qux (as expected).
      foo = super.interpreter27Packages.foo.override { libxml = null; };
    };
  })

you can write

  (self: super: { _merge_interpreter27Packages = true; })
  (self: super: {
    interpreter27Packages.foo = super.interpreter27Packages.foo.override { libxml = null; };
  })

@tomberek as I understand it, there are 2 problems here:

1. Increased memory usage during evaluation (hm, how much?)
2. People will start using it, so in case Nixpkgs later decides that @Ericson2314 approach is better, it will be harder to deprecate this

But I'd like to merge this as is. It helps a bit with overrides in overlays.

This pull request has been mentioned on Nix community. There might be relevant details there:

https://discourse.nixos.org/t/wip-rfc-a-new-nixpkgs-frontend-for-language-infrastructures/3447/9

A year later and I had another look at this, because I need a way to override the Nix packages set from within itself.

@nbp, @Ericson2314 and I had a brief discussion last year on IRC as well about this PR
https://logs.nix.samueldr.com/nixos-dev/2018-07-31#1433188;

The main issue I still have is not having a good way to refer to packages in the main set pkgs. We need a solution there. @nbp proposed renaming e.g. these packages by prepending _ or suffixing '.

Is there a target when the pending review will take place?

I marked this as stale due to inactivity. → More info

Still relevant to me.

Relevant to no one else. I will close this.
Feel free to restart.

This is relevant for #43755.

This is relevant for #43755.

A five-year-old PR stalled for two years - with merge conflicts! Hum...

Edit: I am not doubting on the technical merits of those huge modifications.

However, they are very old: five years, ten releases, lots of new packages inbetween.

A cleaner slate is imperative (pun not intended) in order to get such code in shape.

Also, anyone is free to open this PR or restart and link it.