Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP]: PostgreSQL declarative databases #72365

Closed
wants to merge 2 commits into from

Conversation

talyz
Copy link
Contributor

@talyz talyz commented Oct 31, 2019

This is currently a prototype for declarative databases for the services.postgresql module: the services.postgresql.ensureDatabases option currently leaves databases behind when they're removed from the list. This instead removes them when they're remove from the list. It also adds additional functionality to the services.postgresql.ensureDatabases option, so that owner and extensions can be specified; these, along with services.postgresql.ensureUsers should be made properly declarative in the future if we decide to go this route.

talyz added 2 commits October 31, 2019 08:52
Extend the functionality of ensureDatabases. Each database is
specified as a set with the attributes name, owner and extensions. If
a string is specified, it will be coerced to a set with the attr 'name'
set to the value of the string, for backward compatibility.

- If owner is defined when the database is created, the specified
  user will be created if it doesn't exist, and the database will be
  owned by the specified user.

- If owner is changed, the specified user will be created if it
  doesn't exist, the database owner will be changed, and objects owned
  by the old user will be reassigned to the new one.

- If extensions are specified they will be added to the database.
@ofborg ofborg bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (update) This PR changes an existing module in `nixos/` labels Oct 31, 2019
@talyz talyz mentioned this pull request Oct 31, 2019
10 tasks
@aanderse
Copy link
Member

I would suggest that if a change like this was accepted it has to have the equivalent of users.mutableUsers.

@ofborg ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux labels Oct 31, 2019
@talyz
Copy link
Contributor Author

talyz commented Oct 31, 2019

@aanderse Yeah, I was thinking something similar, since commenting out a piece of code or disabling a service could have potentially devastating effects. I also think we should do cleanup for services in general if we go through with this: with this, disabling a module using ensureDatabases will clean up the database, but not other state tied to the service. Also, disabling the postgresql module won't clean up any state.

@pacien
Copy link
Contributor

pacien commented Nov 1, 2019

Wouldn't switching from one generation to a previous one which doesn't have some module enabled cause unintended data loss? If so, this seems dangerous to me.

@talyz
Copy link
Contributor Author

talyz commented Nov 1, 2019

It would indeed and it would be intended behaviour. We would probably want a switch to toggle this, as discussed above, and also prompt people to back up their data if this is turned on.

Ensure the existence of the listed databases with their
respective owner and extensions.
This option will never delete existing databases; if the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needs to be adjusted

@florianjacob
Copy link
Contributor

In general, it's not really ensureDatabases anymore if it's fully declarative, at least not with my original intention of the prefix, and it doesn't fit to e.g. the config.users.users option then. Hadn't have the case yet, as I did not come up with a declarative implementation that would make sense to me yet. With the databases, I intentionally refrained from deleting user data without manual intervention under any circumstance, but as an option you have to explicitly opt in to the possibility, I guess it would be fine. Personally, I don't drop databases often enough for that I can't afford the manual intervention. Being fully declarative at the owner and extensions module is much less destructive, and definitely a good addition, as people probably seldomly modify those manually.

@talyz
Copy link
Contributor Author

talyz commented Nov 30, 2019

@florianjacob Sorry for the late reply. Anyway, I agree. To begin with, I wanted to show this as a possible way forward if we want stuff like this to be fully declarative. It's by no means complete - I decided to ignore naming, documentation, users and extensions for now; before I put in all the work it would take to get this complete I wanted to know if there was any interest.

Maybe @grahamc and @adisbladis could chime in?

for more details.
'';
};
extensions = mkOption {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This option seems to be on the wrong level.
Extensions in Postgres do not exist on a database level.
In fact, you cannot even have the same extension loaded in different versions on different schemas, even though the documentation might suggest that.
Because of that, I think this option belongs on the service layer, not on a database specific layer in the options.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, that doesn't sound right. I'd have to test this myself to confirm, but if that's true, the documentation is plain wrong. Refer to https://www.postgresql.org/docs/current/sql-createextension.html - the first sentence is CREATE EXTENSION loads a new extension into the current database.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not impossible I am remembering this from an older version, but I have this nagging feeling that something was iffy here. Best to test, I agree.

@aanderse
Copy link
Member

There has been some discussion between several community members on this topic and the hope is that after some more discussion an agreement can be had with a solution that is amicable to all. More to come.

@danbst
Copy link
Contributor

danbst commented Dec 17, 2019

I have an idea 💡 on how to solve this "ensure*" in a fine way. Why not create a __nix table in database, given we already have root access there? Maybe even in separate nix schema, to not clutter public.

We can store there all the stuff managed by Nix - state version, list of databases to maintain, list of users to maintain, list of extensions to maintain (cc @kampka).

Then add a new option:

nixObjectRemoval = mkOption {
  type = types.enum [ "allow" "disallow" "ignore" ];
  default = "disallow";
  description = ''
    Remove objects in database (users, databases, extensions) if their corresponding NixOS options are removed.</para><para>
    When <literal>allow</literal> this is done automatically.</para><para>
    When <literal>disallow</literal>, halt rebuild and show objects which should be removed together with their deleting SQL.</para><para>
    When <literal>ignore</literal>, it will only create objects, never remove.
  '';
};

And then just implement all the state management by trying to match existing schemas to this internal database.

The only tricky part is migrating this database from time to time. At least for extensions we can remove the need for migration -- extensions installed directly in nix schema must be present/absent in all other schemas.

@kampka
Copy link
Contributor

kampka commented Dec 17, 2019

I had the same idea and I think it's valid for users and extensions, but I would never delete databases containing user data. You can only go wrong there imo.
We also don't delete directories containing service data when the service is removed but leave this to the user, for good reason I would think.

@talyz
Copy link
Contributor Author

talyz commented Jan 3, 2020

Hm, I'm not sure about putting this data in the database - what's the benefit and what's wrong with the current approach? I like the nixObjectRemoval option idea, though.

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/postgresql-user-permission-setup-for-database-tables-access/5897/2

@stale
Copy link

stale bot commented Aug 13, 2020

Hello, I'm a bot and I thank you in the name of the community for your contributions.

Nixpkgs is a busy repository, and unfortunately sometimes PRs get left behind for too long. Nevertheless, we'd like to help committers reach the PRs that are still important. This PR has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

If this is still important to you and you'd like to remove the stale label, we ask that you leave a comment. Your comment can be as simple as "still important to me". But there's a bit more you can do:

If you received an approval by an unprivileged maintainer and you are just waiting for a merge, you can @ mention someone with merge permissions and ask them to help. You might be able to find someone relevant by using Git blame on the relevant files, or via GitHub's web interface. You can see if someone's a member of the nixpkgs-committers team, by hovering with the mouse over their username on the web interface, or by searching them directly on the list.

If your PR wasn't reviewed at all, it might help to find someone who's perhaps a user of the package or module you are changing, or alternatively, ask once more for a review by the maintainer of the package/module this is about. If you don't know any, you can use Git blame on the relevant files, or GitHub's web interface to find someone who touched the relevant files in the past.

If your PR has had reviews and nevertheless got stale, make sure you've responded to all of the reviewer's requests / questions. Usually when PR authors show responsibility and dedication, reviewers (privileged or not) show dedication as well. If you've pushed a change, it's possible the reviewer wasn't notified about your push via email, so you can always officially request them for a review, or just @ mention them and say you've addressed their comments.

Lastly, you can always ask for help at our Discourse Forum, or more specifically, at this thread or at #nixos' IRC channel.

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Aug 13, 2020
@talyz
Copy link
Contributor Author

talyz commented Sep 5, 2020

Closing, since the interest in this seems very limited and any potential implementation of this likely would have to be rewritten from scratch.

@talyz talyz closed this Sep 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (update) This PR changes an existing module in `nixos/` 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants