Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

postgresql: reorganize package and its extensions #54319

Merged
merged 5 commits into from Jan 26, 2019
Merged

postgresql: reorganize package and its extensions #54319

merged 5 commits into from Jan 26, 2019

Conversation

danbst
Copy link
Contributor

@danbst danbst commented Jan 19, 2019 •
edited

Motivation for this change

Extracts some useful parts of #38698,
in particular, it's vision that postgresql plugins should be namespaced.

Original approach had several problems:

  • not gonna happen in forseeable future
  • did lots of deprecations
  • was all-in-one solution, which is hard to sell to nixpkgs
  • even that we have postgresqlPackages now, we can't do arbitrary overrides
    to postgresql and plugins. Several required functions were not exported

Here I've fixed all of those problems:

  • deprecates nothing (though plugins were moved now into aliases.nix)
  • this doesn't touch NixOS at all, and doesn't break anything
  • hashes for plugins and PGs are not changed (I hope)
  • no extra fixes to pg itself
  • default PG is not changed
  • plugins and PGs are extensible

Last one is the most interesting thing, because it introduces novel way
to manage XXX withPackages problem. It is novel, but has got lots of
inspiration from existing approaches:

  • python, so we have now postgresql.pkgs.*, postgresql_11.pkgs.*
    which all contain plugins compiled with correct PG.
  • python, so we have now postgresql.withPackages as well
  • in contrast to python, there are no postgresql11Packages, only
    postgresql_11.pkgs
  • Add pkgs.overrideWithScope #44196, so plugins are referenced starting at self-fixpoint.
    This allows override/add plugins with mere // in overlay. This works for
    both postgresqlPackages (overrides are applied to all postgresql_xx.pkgs)
    and postgresql_xx.pkgs (overrides are specific to this postgresql) sets
  • I've made it compatible with proposed mergeable overlays (overlays: (partially) recursive merge #54266)
    however this PR doesn't depend on it
  • last, but not least, postgresql/default.nix is now an overlay! This
    replaces previous callPackages approach with a modern, extensible concept.

Further improvements should go in subsequent PRs! I have a plan to merge @thoughtpolice 's PR part-by-part till 19.03. If nobody explicitly objects current implementation, I'll merge soon.

Things done

I've checked only that hashes didn't change for postgresql_9_6, 10, 11 and default plugins.

  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Assured whether relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@dotlambda dotlambda changed the title postgresql: reorganize package and it's extensions postgresql: reorganize package and its extensions Jan 19, 2019
@danbst danbst mentioned this pull request Jan 19, 2019
Closed
13 tasks
@danbst
Copy link
Contributor Author

danbst commented Jan 19, 2019

thanks, @ofborg

@danbst
Copy link
Contributor Author

danbst commented Jan 19, 2019 •
edited

I'll show a few "override" example, for those who want to do that.

1. Update 1 plugin:

self: super: {
    postgresqlPackages = super.postgresqlPackages // {
        pg_repack = super.postgresqlPackages.pg_repack.overrideAttrs (_: {
            name = "pg_repack-v20181024";
            src = self.fetchzip {
                url = "https://github.com/reorg/pg_repack/archive/923fa2f3c709a506e111cc963034bf2fd127aa00.tar.gz";
                sha256 = "17k6hq9xaax87yz79j773qyigm4fwk8z4zh5cyp6z0sxnwfqxxw5";
            };
        });
    };
}

2. Create 2 PGs, one with plugin update, another with old version of plugin:

    postgresql_production = self.postgresql.withPackages (ps: with ps; [
        pg_repack
    ]);

    postgresql_testing = self.postgresql.withPackages (ps: with ps; [
        pg_repack-v20181024
    ]);

    postgresqlPackages = super.postgresqlPackages // {
        pg_repack-v20181024 = super.postgresqlPackages.pg_repack.overrideAttrs (_: {
            name = "pg_repack-v20181024";
            src = self.fetchzip {
                url = "https://github.com/reorg/pg_repack/archive/923fa2f3c709a506e111cc963034bf2fd127aa00.tar.gz";
                sha256 = "17k6hq9xaax87yz79j773qyigm4fwk8z4zh5cyp6z0sxnwfqxxw5";
            };
        });
    };

3. Same, but lock plugins for one of postgresqls in such a way, that other overlays don't affect this. In fact, it is still possible to override, but becomes more tricky

    postgresql_production' = self.postgresql.override {
        this = self.postgresql_production';
        # 'extends' provide a pristine view of plugins, with whitelist overrides 
        postgresqlPackages = self.postgresql.pkgs.extend (_: plugins: {
            pg_similarity = plugins.pg_similarity.override { gcc = self.gcc8; };
        });
    };
    postgresql_production = self.postgresql_production'.withPackages (ps: with ps; [
        pg_repack  # default
        pg_similarity  # extended (overriden)
    ]);

    postgresql_testing = self.postgresql.withPackages (ps: with ps; [
        pg_repack  # overriden
        pg_similarity # default
    ]);

    postgresqlPackages = super.postgresqlPackages // {
        pg_repack = super.postgresqlPackages.pg_repack.overrideAttrs (_: {
            name = "pg_repack-v20181024";
            src = self.fetchzip {
                url = "https://github.com/reorg/pg_repack/archive/923fa2f3c709a506e111cc963034bf2fd127aa00.tar.gz";
                sha256 = "17k6hq9xaax87yz79j773qyigm4fwk8z4zh5cyp6z0sxnwfqxxw5";
            };
        });
    };

4. Override plugins only for postgresql 11

    postgresql_testing = self.postgresql.withPackages (ps: with ps; [
        pg_repack
        pg_similarity
    ]);

    postgresql_11 = super.postgresql_11.override { postgresqlPackages = self.postgresql_11.pkgs; } // {
        pkgs = super.postgresql_11.pkgs // {
            pg_repack = super.postgresqlPackages.pg_repack.overrideAttrs (_: {
                name = "pg_repack-v20181024";
                src = self.fetchzip {
                    url = "https://github.com/reorg/pg_repack/archive/923fa2f3c709a506e111cc963034bf2fd127aa00.tar.gz";
                    sha256 = "17k6hq9xaax87yz79j773qyigm4fwk8z4zh5cyp6z0sxnwfqxxw5";
                };
            });
        };
    };
    pg11_with_plugins = self.postgresql_11.withPackages (ps: with ps; [
        pg_repack
        pg_similarity
    ]);

@nbp does this look sane?
EDIT: these examples are no longer valid.

@ingenieroariel
Copy link
Member

On this branch I tried both pkgs.postgresqlPackages.postgis and (postgis.override { postgresql = postgresql} ) and they failed.

The first one told me postgis did not exist and the second one gave me the following error:

pdfsdk_headers.h:53:10: fatal error: goo/gtypes.h: No such file or directory

On thoughtpolice's PR the first version works, on 18.09, the second version works.

@danbst
Copy link
Contributor Author

danbst commented Jan 19, 2019

@ingenieroariel that may be true. I did port only a little part here, and never touched postgis. See my plan #38698 (comment), so fixing postgis comes next after this change.

@danbst danbst changed the title postgresql: reorganize package and its extensions [WIP] postgresql: reorganize package and its extensions Jan 20, 2019
@danbst
Copy link
Contributor Author

danbst commented Jan 20, 2019

this has to be rebased over @thoughtpolice changes and fixes for postgis. Didn't expect problems, so developed yet another extender function

@danbst danbst force-pushed the postgresql-packages-extended branch from 83f8de5 to e1d1275 Compare January 20, 2019 22:51
@danbst
postgresql: reorganize package and it's extensions
869bd97 
Extracts some useful parts of NixOS#38698,
in particular, it's vision that postgresql plugins should be namespaced.

Original approach had several problems:
- not gonna happen in forseeable future
- did lots of deprecations
- was all-in-one solution, which is hard to sell to nixpkgs
- even that we have postgresqlPackages now, we can't do arbitrary overrides
  to postgresql and plugins. Several required functions were not exported

Here I've fixed all of those problems:
- deprecates nothing (though plugins were moved now into `aliases.nix`)
- this doesn't touch NixOS at all, and doesn't break anything
- hashes for plugins and PGs are not changed (I hope)
- no extra fixes to pg itself
- default PG is not changed
- plugins and PGs are extensible

Last one is the most interesting thing, because it introduces novel way
to manage `XXX withPackages` problem. It is novel, but has got lots of
inspiration from existing approaches:
- python, so we have now `postgresql.pkgs.*`, `postgresql_11.pkgs.*`
  which all contain plugins compiled with correct PG.
- python, so we have now `postgresql.withPackages` as well
- in contrast to python, there are no `postgresql11Packages`, only
  `postgresql_11.pkgs`
- NixOS#44196, so plugins are referenced starting at self-fixpoint.
  This allows override/add plugins with mere `//` in overlay. This works for
  both `postgresqlPackages` (overrides are applied to all postgresql_xx.pkgs)
  and `postgresql_xx.pkgs` (overrides are specific to this postgresql) sets
- I've made it compatible with proposed mergeable overlays (NixOS#54266)
  however this PR doesn't depend on it
- last, but not least, `postgresql/default.nix` is now an overlay! This
  replaces previous `callPackages` approach with a modern, extensible concept.
@danbst
postgresql: redo extensible packages via overlays
a88af1c 
Previous approach turned out to be awful. It was impossible to perform
deep override.

This time I didn't invent bycicles and used overlays for subpackages.

Big change is that now overriding `postgresqlPackages` doesn't override
all other package sets. But `postgresqlPackages` are now also available
as an overlay! So you can get that, reorganize whatever you want and then
attach to some postgresql.
@danbst
fix postgresql test
8a9e8f4
@danbst
rebase to master
9e8f256
@danbst danbst force-pushed the postgresql-packages-extended branch from e1d1275 to 9e8f256 Compare January 20, 2019 23:07
@danbst danbst changed the title [WIP] postgresql: reorganize package and its extensions postgresql: reorganize package and its extensions Jan 20, 2019
@danbst
Copy link
Contributor Author

danbst commented Jan 20, 2019

Removed all my "novelties" and used plain overlays instead. No more fancy "override with default", but at least overrides in overlays work predictably. Ability to override with // is preserved, and it would be even more awesome after #44196 and #54266

@danbst danbst mentioned this pull request Jan 21, 2019
Merged
10 tasks
@nixos-discourse
Copy link

This pull request has been mentioned on Nix community. There might be relevant details there:

https://discourse.nixos.org/t/heads-up-upcoming-changes-to-the-postgresql-infrastructure-and-nixos-module/297/5

@danbst danbst added this to the 19.03 milestone Jan 25, 2019
@danbst danbst merged commit 8e985dc into NixOS:master Jan 26, 2019
@danbst danbst deleted the postgresql-packages-extended branch January 26, 2019 19:16
@danbst danbst mentioned this pull request Jul 16, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@thoughtpolice thoughtpolice Awaiting requested review from thoughtpolice thoughtpolice is a code owner

Assignees
No one assigned
Labels
0.kind: enhancement 6.topic: nixos 8.has: clean-up 8.has: package (new) 10.rebuild-darwin: 1-10 10.rebuild-linux: 11-100
Projects
None yet
Milestone
19.03
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@danbst @ingenieroariel @nixos-discourse @GrahamcOfBorg